diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 02a16fd..0f73191 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -26,11 +26,15 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip - pip install -e ".[dev]" + pip install -e "core[dev]" + pip install -e plugins/communication_protocols/cli[dev] + pip install -e plugins/communication_protocols/http[dev] + pip install -e plugins/communication_protocols/mcp[dev] + pip install -e plugins/communication_protocols/text[dev] - name: Run tests with pytest run: | - pytest tests/ --doctest-modules --junitxml=junit/test-results.xml --cov=src/utcp --cov-report=xml --cov-report=html + pytest core/tests/ plugins/communication_protocols/cli/tests/ plugins/communication_protocols/http/tests/ plugins/communication_protocols/mcp/tests/ plugins/communication_protocols/text/tests/ --doctest-modules --junitxml=junit/test-results.xml --cov=core/src/utcp --cov-report=xml --cov-report=html - name: Upload coverage reports to Codecov uses: codecov/codecov-action@v3 diff --git a/README.md b/README.md index e86d369..43465ca 100644 --- a/README.md +++ b/README.md @@ -1,70 +1,221 @@ -# Universal Tool Calling Protocol (UTCP) +# Universal Tool Calling Protocol (UTCP) 1.0.0 [![Follow Org](https://img.shields.io/github/followers/universal-tool-calling-protocol?label=Follow%20Org&logo=github)](https://github.com/universal-tool-calling-protocol) [![PyPI Downloads](https://static.pepy.tech/badge/utcp)](https://pepy.tech/projects/utcp) [![License](https://img.shields.io/github/license/universal-tool-calling-protocol/python-utcp)](https://github.com/universal-tool-calling-protocol/python-utcp/blob/main/LICENSE) [![CDTM S23](https://img.shields.io/badge/CDTM-S23-0b84f3)](https://cdtm.com/) - ## Introduction -The Universal Tool Calling Protocol (UTCP) is a modern, flexible, and scalable standard for defining and interacting with tools across a wide variety of communication protocols. It is designed to be easy to use, interoperable, and extensible, making it a powerful choice for building and consuming tool-based services. +The Universal Tool Calling Protocol (UTCP) is a modern, flexible, and scalable standard for defining and interacting with tools across a wide variety of communication protocols. UTCP 1.0.0 introduces a modular core with a plugin-based architecture, making it more extensible, testable, and easier to package. -In contrast to other protocols like MCP, UTCP places a strong emphasis on: +In contrast to other protocols, UTCP places a strong emphasis on: * **Scalability**: UTCP is designed to handle a large number of tools and providers without compromising performance. -* **Interoperability**: With support for a wide range of provider types (including HTTP, WebSockets, gRPC, and even CLI tools), UTCP can integrate with almost any existing service or infrastructure. +* **Extensibility**: A pluggable architecture allows developers to easily add new communication protocols, tool storage mechanisms, and search strategies without modifying the core library. +* **Interoperability**: With a growing ecosystem of protocol plugins (including HTTP, SSE, CLI, and more), UTCP can integrate with almost any existing service or infrastructure. * **Ease of Use**: The protocol is built on simple, well-defined Pydantic models, making it easy for developers to implement and use. ![MCP vs. UTCP](https://github.com/user-attachments/assets/3cadfc19-8eea-4467-b606-66e580b89444) +## New Architecture in 1.0.0 +UTCP has been refactored into a core library and a set of optional plugins. -## Usage Examples +### Core Package (`utcp`) -These examples illustrate the core concepts of the UTCP client and server. They are not designed to be a single, runnable example. +The `utcp` package provides the central components and interfaces: +* **Data Models**: Pydantic models for `Tool`, `CallTemplate`, `UtcpManual`, and `Auth`. +* **Pluggable Interfaces**: + * `CommunicationProtocol`: Defines the contract for protocol-specific communication (e.g., HTTP, CLI). + * `ConcurrentToolRepository`: An interface for storing and retrieving tools with thread-safe access. + * `ToolSearchStrategy`: An interface for implementing tool search algorithms. + * `VariableSubstitutor`: Handles variable substitution in configurations. + * `ToolPostProcessor`: Allows for modifying tool results before they are returned. +* **Default Implementations**: + * `UtcpClient`: The main client for interacting with the UTCP ecosystem. + * `InMemToolRepository`: An in-memory tool repository with asynchronous read-write locks. + * `TagAndDescriptionWordMatchStrategy`: An improved search strategy that matches on tags and description keywords. -> **Note:** For complete, end-to-end runnable examples, please refer to the `examples/` directory in this repository. +### Protocol Plugins -### 1. Using the UTCP Client +Communication protocols are now separate, installable packages. This keeps the core lean and allows users to install only the protocols they need. +* `utcp-http`: Supports HTTP, SSE, and streamable HTTP, plus an OpenAPI converter. +* `utcp-cli`: For wrapping local command-line tools. +* `utcp-mcp`: For interoperability with the Model Context Protocol (MCP). +* `utcp-text`: For reading text files. +* `utcp-socket`: Scaffolding for TCP and UDP protocols. (Work in progress, requires update) +* `utcp-gql`: Scaffolding for GraphQL. (Work in progress, requires update) + +## Installation + +Install the core library and any required protocol plugins. -Setting up a client is simple. You point it to a `providers.json` file, and it handles the rest. +```bash +# Install the core client and the HTTP plugin +pip install utcp utcp-http + +# Install the CLI plugin as well +pip install utcp-cli +``` -**`providers.json`** +For development, you can install the packages in editable mode from the cloned repository: + +```bash +# Clone the repository +git clone https://github.com/universal-tool-calling-protocol/python-utcp.git +cd python-utcp -This file tells the client where to find one or more UTCP Manuals (providers which return a list of tools). +# Install the core package in editable mode with dev dependencies +pip install -e core[dev] + +# Install a specific protocol plugin in editable mode +pip install -e plugins/communication_protocols/http +``` + +## Migration Guide from 0.x to 1.0.0 + +Version 1.0.0 introduces several breaking changes. Follow these steps to migrate your project. + +1. **Update Dependencies**: Install the new `utcp` core package and the specific protocol plugins you use (e.g., `utcp-http`, `utcp-cli`). +2. **Configuration**: + * **Configuration Object**: `UtcpClient` is initialized with a `UtcpClientConfig` object, dict or a path to a JSON file containing the configuration. + * **Manual Call Templates**: The `providers_file_path` option is removed. Instead of a file path, you now provide a list of `manual_call_templates` directly within the `UtcpClientConfig`. + * **Terminology**: The term `provider` has been replaced with `call_template`, and `provider_type` is now `call_template_type`. + * **Streamable HTTP**: The `call_template_type` `http_stream` has been renamed to `streamable_http`. +3. **Update Imports**: Change your imports to reflect the new modular structure. For example, `from utcp.client.transport_interfaces.http_transport import HttpProvider` becomes `from utcp_http.http_call_template import HttpCallTemplate`. +4. **Tool Search**: If you were using the default search, the new strategy is `TagAndDescriptionWordMatchStrategy`. This is the new default and requires no changes unless you were implementing a custom strategy. +5. **Tool Naming**: Tool names are now namespaced as `manual_name.tool_name`. The client handles this automatically. +6 **Variable Substitution Namespacing**: Variables that are subsituted in different `call_templates`, are first namespaced with the name of the manual with the `_` duplicated. So a key in a tool call template called `API_KEY` from the manual `manual_1` would be converted to `manual__1_API_KEY`. + +## Usage Examples + +### 1. Using the UTCP Client + +**`config.json`** (Optional) + +You can define a comprehensive client configuration in a JSON file. All of these fields are optional. ```json -[ - { - "name": "cool_public_apis", - "provider_type": "http", - "url": "http://utcp.io/public-apis-manual", - "http_method": "GET" - } -] +{ + "variables": { + "openlibrary_URL": "https://openlibrary.org/static/openapi.json" + }, + "load_variables_from": [ + { + "variable_loader_type": "dotenv", + "env_file_path": ".env" + } + ], + "tool_repository": { + "tool_repository_type": "in_memory" + }, + "tool_search_strategy": { + "tool_search_strategy_type": "tag_and_description_word_match" + }, + "manual_call_templates": [ + { + "name": "openlibrary", + "call_template_type": "http", + "http_method": "GET", + "url": "${URL}", + "content_type": "application/json" + }, + ], + "post_processing": [ + { + "tool_post_processor_type": "filter_dict", + "only_include_keys": ["name", "key"], + "only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"] + } + ] +} ``` **`client.py`** -This script initializes the client and calls a tool from the provider defined above. - ```python import asyncio -from utcp.client import UtcpClient +from utcp.utcp_client import UtcpClient +from utcp.data.utcp_client_config import UtcpClientConfig async def main(): - # Create a client instance. It automatically loads providers - # from the specified file path. - client = await UtcpClient.create( - config={"providers_file_path": "./providers.json"} + # The UtcpClient can be created with a config file path, a dict, or a UtcpClientConfig object. + + # Option 1: Initialize from a config file path + # client_from_file = await UtcpClient.create(config="./config.json") + + # Option 2: Initialize from a dictionary + client_from_dict = await UtcpClient.create(config={ + "variables": { + "openlibrary_URL": "https://openlibrary.org/static/openapi.json" + }, + "load_variables_from": [ + { + "variable_loader_type": "dotenv", + "env_file_path": ".env" + } + ], + "tool_repository": { + "tool_repository_type": "in_memory" + }, + "tool_search_strategy": { + "tool_search_strategy_type": "tag_and_description_word_match" + }, + "manual_call_templates": [ + { + "name": "openlibrary", + "call_template_type": "http", + "http_method": "GET", + "url": "${URL}", + "content_type": "application/json" + } + ], + "post_processing": [ + { + "tool_post_processor_type": "filter_dict", + "only_include_keys": ["name", "key"], + "only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"] + } + ] + }) + + # Option 3: Initialize with a full-featured UtcpClientConfig object + from utcp_http.http_call_template import HttpCallTemplate + from utcp.data.variable_loader import VariableLoaderSerializer + from utcp.interfaces.tool_post_processor import ToolPostProcessorConfigSerializer + + config_obj = UtcpClientConfig( + variables={"openlibrary_URL": "https://openlibrary.org/static/openapi.json"}, + load_variables_from=[ + VariableLoaderSerializer().validate_dict({ + "variable_loader_type": "dotenv", "env_file_path": ".env" + }) + ], + manual_call_templates=[ + HttpCallTemplate( + name="openlibrary", + call_template_type="http", + http_method="GET", + url="${URL}", + content_type="application/json" + ) + ], + post_processing=[ + ToolPostProcessorConfigSerializer().validate_dict({ + "tool_post_processor_type": "filter_dict", + "only_include_keys": ["name", "key"], + "only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"] + }) + ] ) + client = await UtcpClient.create(config=config_obj) - # Call a tool. The name is namespaced: `provider_name.tool_name` + # Call a tool. The name is namespaced: `manual_name.tool_name` result = await client.call_tool( - tool_name="cool_public_apis.example_tool", - arguments={} + tool_name="openlibrary.read_search_authors_json_search_authors_json_get", + tool_args={"q": "J. K. Rowling"} ) print(result) @@ -75,11 +226,39 @@ if __name__ == "__main__": ### 2. Providing a UTCP Manual -Any type of server or service can be exposed as a UTCP tool. The only requirement is that a `UTCPManual` is provided to the client. This manual can be served by the tool itself or, more powerfully, by a third-party registry. This allows for wrapping existing APIs and services that are not natively UTCP-aware. - -Here is a minimal example using FastAPI to serve a `UTCPManual` for a tool: +A `UTCPManual` describes the tools you offer. The key change is replacing `tool_provider` with `call_template`. **`server.py`** + +UTCP decorator version: + +```python +from fastapi import FastAPI +from utcp_http.http_call_template import HttpCallTemplate +from utcp.data.utcp_manual import UtcpManual +from utcp.python_specific_tooling.tool_decorator import utcp_tool + +app = FastAPI() + +# The discovery endpoint returns the tool manual +@app.get("/utcp") +def utcp_discovery(): + return UtcpManual.create_from_decorators(manual_version="1.0.0") + +# The actual tool endpoint +@utcp_tool(tool_call_template=HttpCallTemplate( + name="get_weather", + url=f"https://example.com/api/weather", + http_method="GET" +), tags=["weather"]) +@app.get("/api/weather") +def get_weather(location: str): + return {"temperature": 22.5, "conditions": "Sunny"} +``` + + +No UTCP dependencies server version: + ```python from fastapi import FastAPI @@ -89,11 +268,13 @@ app = FastAPI() @app.get("/utcp") def utcp_discovery(): return { - "version": "1.0", + "manual_version": "1.0.0", + "utcp_version": "1.0.0", "tools": [ { "name": "get_weather", "description": "Get current weather for a location", + "tags": ["weather"], "inputs": { "type": "object", "properties": { @@ -103,11 +284,12 @@ def utcp_discovery(): "outputs": { "type": "object", "properties": { - "temperature": {"type": "number"} + "temperature": {"type": "number"}, + "conditions": {"type": "string"} } }, - "tool_provider": { - "provider_type": "http", + "call_template": { + "call_template_type": "http", "url": "https://example.com/api/weather", "http_method": "GET" } @@ -121,35 +303,20 @@ def get_weather(location: str): return {"temperature": 22.5, "conditions": "Sunny"} ``` -### 3. Full LLM Integration Example +### 3. Full examples -For a complete, end-to-end demonstration of how to integrate UTCP with a Large Language Model (LLM) like OpenAI, see the example in `example/src/full_llm_example/openai_utcp_example.py`. - -This advanced example showcases: -* **Dynamic Tool Discovery**: No hardcoded tool names. The client loads all available tools from the `providers.json` config. -* **Relevant Tool Search**: For each user prompt, it uses `utcp_client.search_tools()` to find the most relevant tools for the task. -* **LLM-Driven Tool Calls**: It instructs the OpenAI model to respond with a custom JSON format to call a tool. -* **Robust Execution**: It parses the LLM's response, executes the tool call via `utcp_client.call_tool()`, and sends the result back to the model for a final, human-readable answer. -* **Conversation History**: It maintains a full conversation history for contextual, multi-turn interactions. - -**To run the example:** -1. Navigate to the `example/src/full_llm_example/` directory. -2. Rename `example.env` to `.env` and add your OpenAI API key. -3. Run `python openai_utcp_example.py`. +You can find full examples in the [examples repository](https://github.com/universal-tool-calling-protocol/utcp-examples). ## Protocol Specification -UTCP is defined by a set of core data models that describe tools, how to connect to them (providers), and how to secure them (authentication). - -### Tool Discovery +### `UtcpManual` and `Tool` Models -For a client to use a tool, it must be provided with a `UtcpManual` object. This manual contains a list of all the tools available from a provider. Depending on the provider type, this manual might be retrieved from a discovery endpoint (like an HTTP URL) or loaded from a local source (like a file for a CLI tool). - -#### `UtcpManual` Model +The `tool_provider` object inside a `Tool` has been replaced by `call_template`. ```json { - "version": "string", + "manual_version": "string", + "utcp_version": "string", "tools": [ { "name": "string", @@ -157,709 +324,171 @@ For a client to use a tool, it must be provided with a `UtcpManual` object. This "inputs": { ... }, "outputs": { ... }, "tags": ["string"], - "tool_provider": { ... } + "call_template": { + "call_template_type": "http", + "url": "https://...", + "http_method": "GET" + } } ] } ``` -* `version`: The version of the UTCP protocol being used. -* `tools`: A list of `Tool` objects. +## Call Template Configuration Examples -### Tool Definition +Configuration examples for each protocol. Remember to replace `provider_type` with `call_template_type`. -Each tool is defined by the `Tool` model. - -#### `Tool` Model - -```json -{ - "name": "string", - "description": "string", - "inputs": { - "type": "object", - "properties": { ... }, - "required": ["string"], - "description": "string", - "title": "string" - }, - "outputs": { ... }, - "tags": ["string"], - "tool_provider": { ... } -} -``` - -* `name`: The name of the tool. -* `description`: A human-readable description of what the tool does. -* `inputs`: A schema defining the input parameters for the tool. This follows a simplified JSON Schema format. -* `outputs`: A schema defining the output of the tool. -* `tags`: A list of tags for categorizing the tool making searching for relevant tools easier. -* `tool_provider`: The `ToolProvider` object that describes how to connect to and use the tool. - -### Authentication - -UTCP supports several authentication methods to secure tool access. The `auth` object within a provider's configuration specifies the authentication method to use. - -#### API Key (`ApiKeyAuth`) - -Authentication using a static API key that can be sent in different locations. - -```json -{ - "auth_type": "api_key", - "api_key": "YOUR_SECRET_API_KEY", - "var_name": "X-API-Key", - "location": "header" -} -``` - -**Key Fields:** -* `api_key`: Your secret API key -* `var_name`: The name of the parameter (header name, query parameter name, or cookie name) -* `location`: Where to send the API key - `"header"` (default), `"query"`, or `"cookie"` - -**Examples:** - -*Header-based API key (most common):* -```json -{ - "auth_type": "api_key", - "api_key": "sk-1234567890abcdef", - "var_name": "Authorization", - "location": "header" -} -``` - -*Query parameter-based API key:* -```json -{ - "auth_type": "api_key", - "api_key": "abc123def456", - "var_name": "api_key", - "location": "query" -} -``` - -*Cookie-based API key:* -```json -{ - "auth_type": "api_key", - "api_key": "session_token_xyz", - "var_name": "auth_token", - "location": "cookie" -} -``` - -#### Basic Auth (`BasicAuth`) - -Authentication using a username and password. - -```json -{ - "auth_type": "basic", - "username": "your_username", - "password": "your_password" -} -``` - -#### OAuth2 (`OAuth2Auth`) - -Authentication using the OAuth2 client credentials flow. The UTCP client will automatically fetch a bearer token from the `token_url` and use it for subsequent requests. - -```json -{ - "auth_type": "oauth2", - "token_url": "https://auth.example.com/token", - "client_id": "your_client_id", - "client_secret": "your_client_secret", - "scope": "read write" -} -``` - -### Providers - -Providers are at the heart of UTCP's flexibility. They define the communication protocol for a given tool. UTCP supports a wide range of provider types: - -* `http`: RESTful HTTP/HTTPS API -* `sse`: Server-Sent Events -* `http_stream`: HTTP Chunked Transfer Encoding -* `cli`: Command Line Interface -* `websocket`: WebSocket bidirectional connection (work in progress) -* `grpc`: gRPC (Google Remote Procedure Call) (work in progress) -* `graphql`: GraphQL query language (work in progress) -* `tcp`: Raw TCP socket -* `udp`: User Datagram Protocol -* `webrtc`: Web Real-Time Communication (work in progress) -* `mcp`: Model Context Protocol (for interoperability) -* `text`: Local text file - -Each provider type has its own specific configuration options. For example, an `HttpProvider` will have a `url` and an `http_method`. - -## Provider Configuration Examples - -Below are examples of how to configure each of the supported provider types in a JSON configuration file. Where possible, the tool discovery endpoint should be `/utcp`. Each tool provider should offer users their json provider configuration for the tool discovery endpoint. - -### HTTP Provider - -For connecting to standard RESTful APIs. +### HTTP Call Template ```json { "name": "my_rest_api", - "provider_type": "http", - "url": "https://api.example.com/utcp", - "http_method": "POST", - "content_type": "application/json", - "headers": { - "User-Agent": "MyApp/1.0" + "call_template_type": "http", // Required + "url": "https://api.example.com/users/{user_id}", // Required + "http_method": "POST", // Required, default: "GET" + "content_type": "application/json", // Optional, default: "application/json" + "auth": { // Optional, example using ApiKeyAuth for a Bearer token. The client must prepend "Bearer " to the token. + "auth_type": "api_key", + "api_key": "Bearer $API_KEY", // Required + "var_name": "Authorization", // Optional, default: "X-Api-Key" + "location": "header" // Optional, default: "header" }, - "body_field": "LLM_generated_param_to_be_sent_as_body", - "header_fields": ["LLM_generated_param_to_be_sent_as_header"], - "auth": { - "auth_type": "oauth2", - "token_url": "https://api.example.com/oauth/token", - "client_id": "your_client_id", - "client_secret": "your_client_secret" - } -} -``` - -**Key HttpProvider Fields:** -* `http_method`: HTTP method - `"GET"`, `"POST"`, `"PUT"`, `"DELETE"`, `"PATCH"` (default: `"GET"`) -* `url`: The endpoint URL (supports path parameters with `{param}` syntax) -* `content_type`: Content-Type header for request body (default: `"application/json"`) -* `headers`: Static headers to include in all requests -* `body_field`: Name of the input field to use as request body (default: `"body"`) -* `header_fields`: List of input fields to send as request headers -* `auth`: Authentication configuration - -#### Automatic OpenAPI Conversion - -UTCP simplifies integration with existing web services by automatically converting OpenAPI v3 specifications into UTCP tools. Instead of pointing to a `UtcpManual`, the `url` for an `http` provider can point directly to an OpenAPI JSON specification. The `OpenApiConverter` handles this conversion automatically, making it seamless to integrate thousands of existing APIs. - -```json -{ - "name": "open_library_api", - "provider_type": "http", - "url": "https://openlibrary.org/dev/docs/api/openapi.json" -} -``` - -When the client registers this provider, it will fetch the OpenAPI spec from the URL, convert all defined endpoints into UTCP `Tool` objects, and make them available for searching and calling. - -#### URL Path Parameters - -HTTP-based providers (HTTP, SSE, HTTP Stream) support dynamic URL path parameters that can be substituted from tool arguments. This enables integration with RESTful APIs that use path-based resource identification. - -**URL Template Format:** -Path parameters are specified in the URL using curly braces: `{parameter_name}` - -**Example:** -```json -{ - "name": "openlibrary_api", - "provider_type": "http", - "url": "https://openlibrary.org/api/volumes/brief/{key_type}/{value}.json", - "http_method": "GET" -} -``` - -**How it works:** -1. When calling a tool, parameters matching the path parameter names are extracted from the tool arguments -2. These parameters are substituted into the URL template -3. The used parameters are removed from the arguments (so they don't become query parameters) -4. Any remaining arguments become query parameters - -**Example usage:** -```python -# Tool call arguments -arguments = { - "key_type": "isbn", - "value": "9780140328721", - "format": "json" -} - -# Results in URL: https://openlibrary.org/api/volumes/brief/isbn/9780140328721.json?format=json -``` - -**Multiple Path Parameters:** -URLs can contain multiple path parameters: -```json -{ - "url": "https://api.example.com/users/{user_id}/posts/{post_id}/comments/{comment_id}" -} -``` - -**Error Handling:** -- If a required path parameter is missing from the tool arguments, an error is raised -- All path parameters must be provided for the tool call to succeed - -### Server-Sent Events (SSE) Provider - -For tools that stream data using SSE. The `url` should point to the discovery endpoint. - -```json -{ - "name": "live_updates_service", - "provider_type": "sse", - "url": "https://api.example.com/stream", - "event_type": "message", - "reconnect": true, - "retry_timeout": 30000, - "headers": { - "Accept": "text/event-stream" + "headers": { // Optional + "X-Custom-Header": "value" }, - "body_field": null, - "header_fields": ["LLM_generated_param_to_be_sent_as_header"], - "auth": { - "auth_type": "api_key", - "api_key": "your_api_key", - "var_name": "Authorization", - "location": "header" - } + "body_field": "body", // Optional, default: "body" + "header_fields": ["user_id"] // Optional } ``` -**Key SSEProvider Fields:** -* `url`: The SSE endpoint URL (supports path parameters) -* `event_type`: Filter for specific SSE event types (optional) -* `reconnect`: Whether to automatically reconnect on disconnect (default: `true`) -* `retry_timeout`: Retry timeout in milliseconds (default: `30000`) -* `headers`: Static headers for the SSE connection -* `body_field`: Input field for connection request body (optional) -* `header_fields`: Input fields to send as headers for initial connection - -### HTTP Stream Provider - -For tools that use HTTP chunked transfer encoding to stream data. The `url` should point to the discovery endpoint. +### SSE (Server-Sent Events) Call Template ```json { - "name": "streaming_data_source", - "provider_type": "http_stream", - "url": "https://api.example.com/stream", - "http_method": "POST", - "content_type": "application/octet-stream", - "chunk_size": 4096, - "timeout": 60000, - "headers": { - "Accept": "application/octet-stream" - }, - "body_field": "data", - "header_fields": ["LLM_generated_param_to_be_sent_as_header"], - "auth": { + "name": "my_sse_stream", + "call_template_type": "sse", // Required + "url": "https://api.example.com/events", // Required + "event_type": "message", // Optional + "reconnect": true, // Optional, default: true + "retry_timeout": 30000, // Optional, default: 30000 (ms) + "auth": { // Optional, example using BasicAuth "auth_type": "basic", - "username": "your_username", - "password": "your_password" - } -} -``` - -**Key StreamableHttpProvider Fields:** -* `http_method`: HTTP method - `"GET"` or `"POST"` (default: `"GET"`) -* `url`: The streaming endpoint URL (supports path parameters) -* `content_type`: Content-Type for streaming data (default: `"application/octet-stream"`, also supports `"application/x-ndjson"`, `"application/json"`) -* `chunk_size`: Size of chunks in bytes (default: `4096`) -* `timeout`: Timeout in milliseconds (default: `60000`) -* `headers`: Static headers for the stream connection -* `body_field`: Input field for request body (optional) -* `header_fields`: Input fields to send as headers - -### CLI Provider - -For wrapping local command-line tools. - -```json -{ - "name": "my_cli_tool", - "provider_type": "cli", - "command_name": "my-command --utcp", - "env_vars": { - "MY_API_KEY": "${API_KEY}", - "DEBUG": "1" + "username": "${USERNAME}", // Required + "password": "${PASSWORD}" // Required }, - "working_dir": "/path/to/working/directory" -} -``` - -**Key CliProvider Fields:** -* `command_name`: The command to execute (should support UTCP discovery) -* `env_vars`: Environment variables to set when executing (optional) -* `working_dir`: Working directory for command execution (optional) -* `auth`: Always `null` (CLI tools don't use UTCP auth) - -### WebSocket Provider (work in progress) - -For tools that communicate over a WebSocket connection. - -```json -{ - "name": "realtime_chat_service", - "provider_type": "websocket", - "url": "wss://api.example.com/socket" -} -``` - -### gRPC Provider (work in progress) - -For connecting to gRPC services. - -```json -{ - "name": "my_grpc_service", - "provider_type": "grpc", - "host": "grpc.example.com", - "port": 50051, - "service_name": "MyService", - "method_name": "MyMethod", - "use_ssl": true -} -``` - -### GraphQL Provider (work in progress) - -For interacting with GraphQL APIs. - -```json -{ - "name": "my_graphql_api", - "provider_type": "graphql", - "url": "https://api.example.com/graphql", - "operation_type": "query", - "operation_name": "GetUserData", - "headers": { - "Content-Type": "application/json" + "headers": { // Optional + "X-Client-ID": "12345" }, - "header_fields": ["LLM_generated_param_to_be_sent_as_header"], - "auth": { - "auth_type": "oauth2", - "token_url": "https://api.example.com/oauth/token", - "client_id": "graphql_client", - "client_secret": "secret_123" - } -} -``` - -**Key GraphQLProvider Fields:** -* `url`: The GraphQL endpoint URL -* `operation_type`: Type of GraphQL operation - `"query"`, `"mutation"`, `"subscription"` (default: `"query"`) -* `operation_name`: Name of the GraphQL operation (optional) -* `headers`: Static headers for GraphQL requests -* `header_fields`: Input fields to send as headers - -### TCP Provider - -For TCP socket communication. Supports multiple framing strategies, JSON and text-based request formats, and configurable response handling. - -**Basic Example:** -```json -{ - "name": "tcp_service", - "provider_type": "tcp", - "host": "localhost", - "port": 12345, - "timeout": 30000, - "request_data_format": "json", - "framing_strategy": "stream", - "response_byte_format": "utf-8" -} -``` - -**Key TCP Provider Fields:** - -* `host`: The hostname or IP address of the TCP server -* `port`: The TCP port number -* `timeout`: Timeout in milliseconds (default: 30000) -* `request_data_format`: Either `"json"` for structured data or `"text"` for template-based formatting (default: `"json"`) -* `request_data_template`: Template string for text format with `UTCP_ARG_argname_UTCP_ARG` placeholders -* `response_byte_format`: Encoding for response bytes - `"utf-8"`, `"ascii"`, etc., or `null` for raw bytes (default: `"utf-8"`) -* `framing_strategy`: Message framing strategy: `"stream"`, `"length_prefix"`, `"delimiter"`, or `"fixed_length"` (default: `"stream"`) -* `length_prefix_bytes`: For length-prefix framing: 1, 2, 4, or 8 bytes (default: 4) -* `length_prefix_endian`: For length-prefix framing: `"big"` or `"little"` (default: `"big"`) -* `message_delimiter`: For delimiter framing: delimiter string like `"\n"`, `"\r\n"`, `"\x00"` (default: `"\x00"`) -* `fixed_message_length`: For fixed-length framing: exact message length in bytes -* `max_response_size`: For stream framing: maximum bytes to read (default: 65536) - -**Length-Prefix Framing Example:** -```json -{ - "name": "binary_tcp_service", - "provider_type": "tcp", - "host": "192.168.1.50", - "port": 8080, - "framing_strategy": "length_prefix", - "length_prefix_bytes": 4, - "length_prefix_endian": "big", - "request_data_format": "json", - "response_byte_format": "utf-8" -} -``` - -**Delimiter Framing Example:** -```json -{ - "name": "line_based_tcp_service", - "provider_type": "tcp", - "host": "tcp.example.com", - "port": 9999, - "framing_strategy": "delimiter", - "message_delimiter": "\n", - "request_data_format": "text", - "request_data_template": "GET UTCP_ARG_resource_UTCP_ARG", - "response_byte_format": "ascii" -} -``` - -**Fixed-Length Framing Example:** -```json -{ - "name": "fixed_protocol_service", - "provider_type": "tcp", - "host": "legacy.example.com", - "port": 7777, - "framing_strategy": "fixed_length", - "fixed_message_length": 1024, - "request_data_format": "text", - "response_byte_format": null + "body_field": null, // Optional + "header_fields": [] // Optional } ``` -### UDP Provider +### Streamable HTTP Call Template -For UDP socket communication. Supports both JSON and text-based request formats with configurable response handling. +Note the name change from `http_stream` to `streamable_http`. ```json { - "name": "udp_telemetry_service", - "provider_type": "udp", - "host": "localhost", - "port": 54321, - "timeout": 30000, - "request_data_format": "json", - "number_of_response_datagrams": 1, - "response_byte_format": "utf-8" + "name": "streaming_data_source", + "call_template_type": "streamable_http", // Required + "url": "https://api.example.com/stream", // Required + "http_method": "POST", // Optional, default: "GET" + "content_type": "application/octet-stream", // Optional, default: "application/octet-stream" + "chunk_size": 4096, // Optional, default: 4096 + "timeout": 60000, // Optional, default: 60000 (ms) + "auth": null, // Optional + "headers": {}, // Optional + "body_field": "data", // Optional + "header_fields": [] // Optional } ``` -**Key UDP Provider Fields:** - -* `host`: The hostname or IP address of the UDP server -* `port`: The UDP port number -* `timeout`: Timeout in milliseconds (default: 30000) -* `request_data_format`: Either `"json"` for structured data or `"text"` for template-based formatting (default: `"json"`) -* `request_data_template`: Template string for text format with `UTCP_ARG_argname_UTCP_ARG` placeholders -* `number_of_response_datagrams`: Number of UDP response packets to expect (default: 0 for no response) -* `response_byte_format`: Encoding for response bytes - `"utf-8"`, `"ascii"`, etc., or `null` for raw bytes (default: `"utf-8"`) +### CLI Call Template -**Text Format Example:** ```json { - "name": "legacy_udp_service", - "provider_type": "udp", - "host": "192.168.1.100", - "port": 9999, - "request_data_format": "text", - "request_data_template": "CMD:UTCP_ARG_command_UTCP_ARG;VALUE:UTCP_ARG_value_UTCP_ARG", - "number_of_response_datagrams": 2, - "response_byte_format": "ascii" + "name": "my_cli_tool", + "call_template_type": "cli", // Required + "command_name": "my-command --utcp", // Required + "env_vars": { // Optional + "MY_VAR": "my_value" + }, + "working_dir": "/path/to/working/directory", // Optional + "auth": null // Optional (always null for CLI) } ``` -### WebRTC Provider (work in progress) - -For peer-to-peer communication using WebRTC. +### Text Call Template ```json { - "name": "p2p_data_transfer", - "provider_type": "webrtc", - "signaling_server": "https://signaling.example.com", - "peer_id": "remote-peer-id" + "name": "my_text_manual", + "call_template_type": "text", // Required + "file_path": "./manuals/my_manual.json", // Required + "auth": null // Optional (always null for Text) } ``` -### MCP Provider +### MCP (Model Context Protocol) Call Template -For interoperability with the Model Context Protocol (MCP). This provider can connect to MCP servers via `stdio` or `http`. - -**HTTP MCP Server Example:** ```json { - "name": "my_mcp_http_service", - "provider_type": "mcp", - "config": { + "name": "my_mcp_server", + "call_template_type": "mcp", // Required + "config": { // Required "mcpServers": { - "my-server": { - "transport": "http", - "url": "http://localhost:8000/mcp" + "server_name": { + "transport": "stdio", + "command": ["python", "-m", "my_mcp_server"] } } }, - "auth": { + "auth": { // Optional, example using OAuth2 "auth_type": "oauth2", - "token_url": "http://localhost:8000/token", - "client_id": "test-client", - "client_secret": "test-secret" + "token_url": "https://auth.example.com/token", // Required + "client_id": "${CLIENT_ID}", // Required + "client_secret": "${CLIENT_SECRET}", // Required + "scope": "read:tools" // Optional } } ``` -**Stdio MCP Server Example:** -```json -{ - "name": "my_mcp_stdio_service", - "provider_type": "mcp", - "config": { - "mcpServers": { - "local-server": { - "transport": "stdio", - "command": "python", - "args": ["-m", "my_mcp_server.main"], - "env": { - "API_KEY": "${MCP_API_KEY}", - "DEBUG": "1" - } - } - } - } -} -``` - -**Key MCPProvider Fields:** -* `config`: MCP configuration object containing server definitions -* `config.mcpServers`: Dictionary of server name to server configuration -* `auth`: OAuth2 authentication (optional, only for HTTP servers) - -**MCP Server Types:** -* **HTTP**: `{"transport": "http", "url": "server_url"}` -* **Stdio**: `{"transport": "stdio", "command": "cmd", "args": [...], "env": {...}}` - -### Text Provider - -For loading tool definitions from a local text file. This is useful for defining a collection of tools that may use various other providers. - -```json -{ - "name": "my_local_tools", - "provider_type": "text", - "file_path": "/path/to/my/tools.json" -} -``` - -**Key TextProvider Fields:** -* `file_path`: Path to the file containing tool definitions (required) -* `auth`: Always `null` (text files don't require authentication) - -**Use Cases:** -- Define tools that produce static output files -- Create tool collections that reference other providers -- Download manuals from a remote server to allow inspection of tools before calling them and guarantee security for high-risk environments - - - -### Authentication - -UTCP supports several authentication methods, which can be configured on a per-provider basis: - -* **API Key**: `ApiKeyAuth` - Authentication using an API key that can be sent in headers, query parameters, or cookies -* **Basic Auth**: `BasicAuth` - Authentication using a username and password -* **OAuth2**: `OAuth2Auth` - Authentication using the OAuth2 client credentials flow with automatic token management - -#### Enhanced Authentication Features - -**Flexible API Key Placement:** -- Headers (most common): `"location": "header"` -- Query parameters: `"location": "query"` -- Cookies: `"location": "cookie"` - -**OAuth2 Automatic Token Management:** -- Supports both body-based and header-based OAuth2 token requests -- Automatic token caching and reuse -- Fallback mechanisms for different OAuth2 server implementations - -**Comprehensive HTTP Transport Support:** -All HTTP-based transports (HTTP, SSE, HTTP Stream) support the full range of authentication methods with proper configuration handling during both tool discovery and tool execution. - -## UTCP Client Architecture - -The Python UTCP client provides a robust and extensible framework for interacting with tool providers. Its architecture is designed around a few key components that work together to manage, execute, and search for tools. - -### Core Components - -* **`UtcpClient`**: The main entry point for interacting with the UTCP ecosystem. It orchestrates the registration of providers, the execution of tools, and the search for available tools. -* **`UtcpClientConfig`**: A Pydantic model that defines the client's configuration. It specifies the path to the providers' configuration file (`providers_file_path`) and how to load sensitive variables (e.g., from a `.env` file using `load_variables_from`). -* **`ClientTransportInterface`**: An abstract base class that defines the contract for all transport implementations (e.g., `HttpClientTransport`, `CliTransport`). Each transport is responsible for the protocol-specific communication required to register and call tools. -* **`ToolRepository`**: An abstract base class that defines the interface for storing and retrieving tools and providers. The default implementation is `InMemToolRepository`, which stores everything in memory. -* **`ToolSearchStrategy`**: An abstract base class for implementing different tool search algorithms. The default is `TagSearchStrategy`, which scores tools based on matching tags and keywords from the tool's description. - -### Initialization and Configuration - -A `UtcpClient` instance is created using the asynchronous `UtcpClient.create()` class method. This method initializes the client with a configuration, a tool repository, and a search strategy. - -```python -import asyncio -from utcp.client import UtcpClient - -async def main(): - # The client automatically loads providers from the path specified in the config - client = await UtcpClient.create( - config={ - "providers_file_path": "/path/to/your/providers.json", - "load_variables_from": [{ - "type": "dotenv", - "env_file_path": ".env" - }] - } - ) - # ... use the client - -asyncio.run(main()) -``` - -During initialization, the client reads the `providers.json` file, substitutes any variables (e.g., `${API_KEY}`), and registers each provider. - -### Tool Management and Execution - -- **Registration**: The `register_tool_provider` method uses the appropriate transport to fetch the tool definitions from a provider and saves them in the `ToolRepository`. -- **Execution**: The `call_tool` method finds the requested tool in the repository, retrieves its provider information, and uses the correct transport to execute the call with the given arguments. Tool names are namespaced by their provider (e.g., `my_api.get_weather`). -- **Deregistration**: Providers can be deregistered, which removes them and their associated tools from the repository. - -### Tool Search - -The `search_tools` method allows you to find relevant tools based on a query. It delegates the search to the configured `ToolSearchStrategy`. - -```python -tools = client.search_tools(query="get current weather in London") -for tool in tools: - print(tool.name, tool.description) -``` - ## Testing -The UTCP client includes comprehensive test suites for all transport implementations. Tests cover functionality, error handling, different configuration options, and edge cases. +The testing structure has been updated to reflect the new core/plugin split. ### Running Tests -To run all tests: +To run all tests for the core library and all plugins: ```bash +# Ensure you have installed all dev dependencies python -m pytest ``` -To run tests for a specific transport (e.g., TCP): +To run tests for a specific package (e.g., the core library): ```bash -python -m pytest tests/client/transport_interfaces/test_tcp_transport.py -v +python -m pytest core/tests/ +``` + +To run tests for a specific plugin (e.g., HTTP): +```bash +python -m pytest plugins/communication_protocols/http/tests/ -v ``` To run tests with coverage: ```bash -python -m pytest --cov=utcp tests/ +python -m pytest --cov=utcp --cov-report=xml ``` ## Build -1. Create a virtual environment (e.g. `conda create --name utcp python=3.10`) and enable it (`conda activate utcp`) -2. Install required libraries (`pip install -r requirements.txt`) -3. `python -m pip install --upgrade pip` -4. `python -m build` -5. `pip install dist/utcp-.tar.gz` (e.g. `pip install dist/utcp-1.0.0.tar.gz`) -# [Contributors](https://www.utcp.io/about) +The build process now involves building each package (`core` and `plugins`) separately if needed, though they are published to PyPI independently. + +1. Create and activate a virtual environment. +2. Install build dependencies: `pip install build`. +3. Navigate to the package directory (e.g., `cd core`). +4. Run the build: `python -m build`. +5. The distributable files (`.whl` and `.tar.gz`) will be in the `dist/` directory. + +## [Contributors](https://www.utcp.io/about) diff --git a/core/pyproject.toml b/core/pyproject.toml new file mode 100644 index 0000000..387af5a --- /dev/null +++ b/core/pyproject.toml @@ -0,0 +1,40 @@ +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" + +[project] +name = "utcp" +version = "1.0.0" +authors = [ + { name = "UTCP Contributors" }, +] +description = "Universal Tool Calling Protocol (UTCP) client library for Python" +readme = "README.md" +requires-python = ">=3.10" +dependencies = [ + "pydantic>=2.0", + "python-dotenv>=1.0", + "tomli>=2.0", +] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "Programming Language :: Python :: 3", + "Operating System :: OS Independent", +] +license = "MPL-2.0" + +[project.optional-dependencies] +dev = [ + "build", + "pytest", + "pytest-asyncio", + "pytest-cov", + "coverage", + "twine", +] + +[project.urls] +Homepage = "https://utcp.io" +Source = "https://github.com/universal-tool-calling-protocol/python-utcp" +Issues = "https://github.com/universal-tool-calling-protocol/python-utcp/issues" \ No newline at end of file diff --git a/core/src/utcp/__init__.py b/core/src/utcp/__init__.py new file mode 100644 index 0000000..d6f3c35 --- /dev/null +++ b/core/src/utcp/__init__.py @@ -0,0 +1,10 @@ +import logging +import sys + +logger = logging.getLogger("utcp") + +if not logger.handlers: # Only add default handler if user didn't configure logging + handler = logging.StreamHandler(sys.stderr) + handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(filename)s:%(lineno)d - %(message)s")) + logger.addHandler(handler) + logger.setLevel(logging.INFO) diff --git a/src/utcp/client/__init__.py b/core/src/utcp/data/__init__.py similarity index 100% rename from src/utcp/client/__init__.py rename to core/src/utcp/data/__init__.py diff --git a/core/src/utcp/data/auth.py b/core/src/utcp/data/auth.py new file mode 100644 index 0000000..875ece9 --- /dev/null +++ b/core/src/utcp/data/auth.py @@ -0,0 +1,33 @@ +"""Authentication schemes for UTCP providers. + +This module defines the authentication models supported by UTCP providers, +including API key authentication, basic authentication, and OAuth2. +""" + +from abc import ABC +from pydantic import BaseModel +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +class Auth(BaseModel, ABC): + """Authentication details for a provider. + + Attributes: + auth_type: The authentication type identifier. + """ + auth_type: str + +class AuthSerializer(Serializer[Auth]): + auth_serializers: dict[str, Serializer[Auth]] = {} + + def to_dict(self, obj: Auth) -> dict: + return AuthSerializer.auth_serializers[obj.auth_type].to_dict(obj) + + def validate_dict(self, obj: dict) -> Auth: + try: + return AuthSerializer.auth_serializers[obj["auth_type"]].validate_dict(obj) + except KeyError: + raise ValueError(f"Invalid auth type: {obj['auth_type']}") + except Exception as e: + raise UtcpSerializerValidationError("Invalid Auth: " + traceback.format_exc()) from e diff --git a/core/src/utcp/data/auth_implementations/__init__.py b/core/src/utcp/data/auth_implementations/__init__.py new file mode 100644 index 0000000..602bf2d --- /dev/null +++ b/core/src/utcp/data/auth_implementations/__init__.py @@ -0,0 +1,12 @@ +from utcp.data.auth_implementations.api_key_auth import ApiKeyAuth, ApiKeyAuthSerializer +from utcp.data.auth_implementations.basic_auth import BasicAuth, BasicAuthSerializer +from utcp.data.auth_implementations.oauth2_auth import OAuth2Auth, OAuth2AuthSerializer + +__all__ = [ + "ApiKeyAuth", + "BasicAuth", + "OAuth2Auth", + "ApiKeyAuthSerializer", + "BasicAuthSerializer", + "OAuth2AuthSerializer" +] diff --git a/core/src/utcp/data/auth_implementations/api_key_auth.py b/core/src/utcp/data/auth_implementations/api_key_auth.py new file mode 100644 index 0000000..47c6ddb --- /dev/null +++ b/core/src/utcp/data/auth_implementations/api_key_auth.py @@ -0,0 +1,42 @@ +from utcp.data.auth import Auth +from utcp.interfaces.serializer import Serializer +from pydantic import Field, ValidationError +from typing import Literal +from utcp.exceptions import UtcpSerializerValidationError + +class ApiKeyAuth(Auth): + """Authentication using an API key. + + The key can be provided directly or sourced from an environment variable. + Supports placement in headers, query parameters, or cookies. + + Attributes: + auth_type: The authentication type identifier, always "api_key". + api_key: The API key for authentication. Values starting with '$' or formatted as '${}' are + treated as an injected variable from environment or configuration. + var_name: The name of the header, query parameter, or cookie that + contains the API key. + location: Where to include the API key (header, query parameter, or cookie). + """ + + auth_type: Literal["api_key"] = "api_key" + api_key: str = Field(..., description="The API key for authentication. Values starting with '$' or formatted as '${}' are treated as an injected variable from environment or configuration. This is the recommended way to provide API keys.") + var_name: str = Field( + "X-Api-Key", description="The name of the header, query parameter, cookie or other container for the API key." + ) + location: Literal["header", "query", "cookie"] = Field( + "header", description="Where to include the API key (header, query parameter, or cookie)." + ) + + +class ApiKeyAuthSerializer(Serializer[ApiKeyAuth]): + def to_dict(self, obj: ApiKeyAuth) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> ApiKeyAuth: + try: + return ApiKeyAuth.model_validate(obj) + except ValidationError as e: + raise UtcpSerializerValidationError(f"Invalid ApiKeyAuth: {e}") from e + except Exception as e: + raise UtcpSerializerValidationError("An unexpected error occurred during ApiKeyAuth validation.") from e diff --git a/core/src/utcp/data/auth_implementations/basic_auth.py b/core/src/utcp/data/auth_implementations/basic_auth.py new file mode 100644 index 0000000..4d09937 --- /dev/null +++ b/core/src/utcp/data/auth_implementations/basic_auth.py @@ -0,0 +1,34 @@ +from utcp.data.auth import Auth +from utcp.interfaces.serializer import Serializer +from pydantic import Field, ValidationError +from typing import Literal +from utcp.exceptions import UtcpSerializerValidationError + +class BasicAuth(Auth): + """Authentication using HTTP Basic Authentication. + + Uses the standard HTTP Basic Authentication scheme with username and password + encoded in the Authorization header. + + Attributes: + auth_type: The authentication type identifier, always "basic". + username: The username for basic authentication. Recommended to use injected variables. + password: The password for basic authentication. Recommended to use injected variables. + """ + + auth_type: Literal["basic"] = "basic" + username: str = Field(..., description="The username for basic authentication.") + password: str = Field(..., description="The password for basic authentication.") + + +class BasicAuthSerializer(Serializer[BasicAuth]): + def to_dict(self, obj: BasicAuth) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> BasicAuth: + try: + return BasicAuth.model_validate(obj) + except ValidationError as e: + raise UtcpSerializerValidationError(f"Invalid BasicAuth: {e}") from e + except Exception as e: + raise UtcpSerializerValidationError("An unexpected error occurred during BasicAuth validation.") from e diff --git a/core/src/utcp/data/auth_implementations/oauth2_auth.py b/core/src/utcp/data/auth_implementations/oauth2_auth.py new file mode 100644 index 0000000..cd178b7 --- /dev/null +++ b/core/src/utcp/data/auth_implementations/oauth2_auth.py @@ -0,0 +1,39 @@ +from utcp.data.auth import Auth +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +from pydantic import Field, ValidationError +from typing import Literal, Optional + + +class OAuth2Auth(Auth): + """Authentication using OAuth2 client credentials flow. + + Implements the OAuth2 client credentials grant type for machine-to-machine + authentication. The client automatically handles token acquisition and refresh. + + Attributes: + auth_type: The authentication type identifier, always "oauth2". + token_url: The URL endpoint to fetch the OAuth2 access token from. Recommended to use injected variables. + client_id: The OAuth2 client identifier. Recommended to use injected variables. + client_secret: The OAuth2 client secret. Recommended to use injected variables. + scope: Optional scope parameter to limit the access token's permissions. + """ + + auth_type: Literal["oauth2"] = "oauth2" + token_url: str = Field(..., description="The URL to fetch the OAuth2 token from.") + client_id: str = Field(..., description="The OAuth2 client ID.") + client_secret: str = Field(..., description="The OAuth2 client secret.") + scope: Optional[str] = Field(None, description="The OAuth2 scope.") + + +class OAuth2AuthSerializer(Serializer[OAuth2Auth]): + def to_dict(self, obj: OAuth2Auth) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> OAuth2Auth: + try: + return OAuth2Auth.model_validate(obj) + except ValidationError as e: + raise UtcpSerializerValidationError(f"Invalid OAuth2Auth: {e}") from e + except Exception as e: + raise UtcpSerializerValidationError("An unexpected error occurred during OAuth2Auth validation.") from e diff --git a/core/src/utcp/data/call_template.py b/core/src/utcp/data/call_template.py new file mode 100644 index 0000000..f79acdd --- /dev/null +++ b/core/src/utcp/data/call_template.py @@ -0,0 +1,75 @@ +"""Provider configurations for UTCP tool providers. + +This module defines the provider models and configurations for all supported +transport protocols in UTCP. Each provider type encapsulates the necessary +configuration to connect to and interact with tools through different +communication channels. + +Supported provider types: + - HTTP: RESTful HTTP/HTTPS APIs + - SSE: Server-Sent Events for streaming + - HTTP Stream: HTTP Chunked Transfer Encoding + - CLI: Command Line Interface tools + - WebSocket: Bidirectional WebSocket connections (WIP) + - gRPC: Google Remote Procedure Call (WIP) + - GraphQL: GraphQL query language + - TCP: Raw TCP socket connections + - UDP: User Datagram Protocol + - WebRTC: Web Real-Time Communication (WIP) + - MCP: Model Context Protocol + - Text: Text file-based providers +""" + +from typing import List, Optional, Union +from pydantic import BaseModel, field_serializer, field_validator, Field +import uuid +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback +from utcp.data.auth import Auth, AuthSerializer + +class CallTemplate(BaseModel): + """Base class for all UTCP tool providers. + + This is the abstract base class that all specific call template implementations + inherit from. It provides the common fields that every provider must have. + + Attributes: + name: Unique identifier for the provider. Defaults to a random UUID hex string. + Should be unique across all providers and recommended to be set to a human-readable name. + Can only contain letters, numbers and underscores. All special characters must be replaced with underscores. + call_template_type: The transport protocol type used by this provider. + """ + + name: str = Field(default_factory=lambda: uuid.uuid4().hex) + call_template_type: str + auth: Optional[Auth] = None + + @field_serializer("auth") + def serialize_auth(self, auth: Optional[Auth]): + if auth is None: + return None + return AuthSerializer().to_dict(auth) + + @field_validator("auth", mode="before") + @classmethod + def validate_auth(cls, v: Optional[Union[Auth, dict]]): + if v is None: + return None + if isinstance(v, Auth): + return v + return AuthSerializer().validate_dict(v) + +class CallTemplateSerializer(Serializer[CallTemplate]): + call_template_serializers: dict[str, Serializer[CallTemplate]] = {} + + def to_dict(self, obj: CallTemplate) -> dict: + return CallTemplateSerializer.call_template_serializers[obj.call_template_type].to_dict(obj) + + def validate_dict(self, obj: dict) -> CallTemplate: + try: + return CallTemplateSerializer.call_template_serializers[obj["call_template_type"]].validate_dict(obj) + except KeyError: + raise ValueError(f"Invalid call template type: {obj['call_template_type']}") + except Exception as e: + raise UtcpSerializerValidationError("Invalid CallTemplate: " + traceback.format_exc()) from e diff --git a/core/src/utcp/data/register_manual_response.py b/core/src/utcp/data/register_manual_response.py new file mode 100644 index 0000000..756b9bd --- /dev/null +++ b/core/src/utcp/data/register_manual_response.py @@ -0,0 +1,10 @@ +from utcp.data.call_template import CallTemplate +from utcp.data.utcp_manual import UtcpManual +from pydantic import BaseModel, Field +from typing import List + +class RegisterManualResult(BaseModel): + manual_call_template: CallTemplate + manual: UtcpManual + success: bool + errors: List[str] = Field(default_factory=list) diff --git a/core/src/utcp/data/tool.py b/core/src/utcp/data/tool.py new file mode 100644 index 0000000..fb85e0d --- /dev/null +++ b/core/src/utcp/data/tool.py @@ -0,0 +1,104 @@ +"""Tool definitions and schema generation for UTCP. + +This module provides the core tool definition models and utilities for +automatic schema generation from Python functions. It supports both +manual tool definitions and decorator-based automatic tool creation. + +Key Components: + - Tool: The main tool definition model + - JSONSchema: JSON Schema for tool inputs and outputs + - ToolContext: Global tool registry +""" + +from typing import Dict, Any, Optional, List +from pydantic import BaseModel, Field, field_serializer, field_validator +from utcp.data.call_template import CallTemplate, CallTemplateSerializer +from utcp.interfaces.serializer import Serializer +from typing import Union +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +JsonType = Union[str, int, float, bool, None, Dict[str, Any], List[Any]] + +class JsonSchema(BaseModel): + schema_: Optional[str] = Field(None, alias="$schema") + id_: Optional[str] = Field(None, alias="$id") + title: Optional[str] = None + description: Optional[str] = None + type: Optional[Union[str, List[str]]] = None + properties: Optional[Dict[str, "JsonSchema"]] = None + items: Optional[Union["JsonSchema", List["JsonSchema"]]] = None + required: Optional[List[str]] = None + enum: Optional[List[JsonType]] = None + const: Optional[JsonType] = None + default: Optional[JsonType] = None + format: Optional[str] = None + additionalProperties: Optional[Union[bool, "JsonSchema"]] = None + pattern: Optional[str] = None + minimum: Optional[float] = None + maximum: Optional[float] = None + minLength: Optional[int] = None + maxLength: Optional[int] = None + + model_config = { + "populate_by_name": True, # replaces allow_population_by_field_name + "extra": "allow" + } + +JsonSchema.model_rebuild() # replaces update_forward_refs() + +class JsonSchemaSerializer(Serializer[JsonSchema]): + def to_dict(self, obj: JsonSchema) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> JsonSchema: + try: + return JsonSchema.model_validate(obj) + except Exception as e: + raise UtcpSerializerValidationError("Invalid JSONSchema: " + traceback.format_exc()) from e + +class Tool(BaseModel): + """Definition of a UTCP tool. + + Represents a callable tool with its metadata, input/output schemas, + and provider configuration. Tools are the fundamental units of + functionality in the UTCP ecosystem. + + Attributes: + name: Unique identifier for the tool, typically in format "provider.tool_name". + description: Human-readable description of what the tool does. + inputs: JSON Schema defining the tool's input parameters. + outputs: JSON Schema defining the tool's return value structure. + tags: List of tags for categorization and search. + average_response_size: Optional hint about typical response size in bytes. + tool_call_template: CallTemplate configuration for accessing this tool. + """ + + name: str + description: str = "" + inputs: JsonSchema = Field(default_factory=JsonSchema) + outputs: JsonSchema = Field(default_factory=JsonSchema) + tags: List[str] = Field(default_factory=list) + average_response_size: Optional[int] = None + tool_call_template: CallTemplate + + @field_serializer("tool_call_template") + def serialize_call_template(self, call_template: CallTemplate): + return CallTemplateSerializer().to_dict(call_template) + + @field_validator("tool_call_template", mode="before") + @classmethod + def validate_call_template(cls, v: Union[CallTemplate, dict]): + if isinstance(v, CallTemplate): + return v + return CallTemplateSerializer().validate_dict(v) + +class ToolSerializer(Serializer[Tool]): + def to_dict(self, obj: Tool) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> Tool: + try: + return Tool.model_validate(obj) + except Exception as e: + raise UtcpSerializerValidationError("Invalid Tool: " + traceback.format_exc()) from e diff --git a/core/src/utcp/data/utcp_client_config.py b/core/src/utcp/data/utcp_client_config.py new file mode 100644 index 0000000..2112f73 --- /dev/null +++ b/core/src/utcp/data/utcp_client_config.py @@ -0,0 +1,119 @@ +from pydantic import BaseModel, Field, field_serializer, field_validator +from typing import Optional, List, Dict, Union, Any +from utcp.data.variable_loader import VariableLoader, VariableLoaderSerializer +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepository, ConcurrentToolRepositoryConfigSerializer +from utcp.interfaces.tool_search_strategy import ToolSearchStrategy, ToolSearchStrategyConfigSerializer +from utcp.data.call_template import CallTemplate, CallTemplateSerializer +from utcp.interfaces.tool_post_processor import ToolPostProcessor, ToolPostProcessorConfigSerializer +import traceback + +class UtcpClientConfig(BaseModel): + """Configuration model for UTCP client setup. + + Provides comprehensive configuration options for UTCP clients including + variable definitions, provider file locations, and variable loading + mechanisms. Supports hierarchical variable resolution with multiple + sources. + + Variable Resolution Order: + 1. Direct variables dictionary + 2. Custom variable loaders (in order) + 3. Environment variables + + Attributes: + variables: Direct variable definitions as key-value pairs. + These take precedence over other variable sources. + providers_file_path: Optional path to a file containing provider + configurations. Supports JSON and YAML formats. + load_variables_from: List of variable loaders to use for + variable resolution. Loaders are consulted in order. + + Example: + ```python + config = UtcpClientConfig( + variables={"MANUAL__NAME_API_KEY_NAME": "$REMAPPED_API_KEY"}, + load_variables_from=[ + VariableLoaderSerializer().validate_dict({"variable_loader_type": "dotenv", "env_file_path": ".env"}) + ], + tool_repository={ + "tool_repository_type": "in_memory" + }, + tool_search_strategy={ + "tool_search_strategy_type": "tag_and_description_word_match" + }, + post_processing=[], + manual_call_templates=[] + ) + ``` + """ + variables: Optional[Dict[str, str]] = Field(default_factory=dict) + load_variables_from: Optional[List[VariableLoader]] = None + tool_repository: ConcurrentToolRepository = Field(default_factory=lambda: ConcurrentToolRepositoryConfigSerializer().validate_dict({"tool_repository_type": ConcurrentToolRepositoryConfigSerializer.default_repository})) + tool_search_strategy: ToolSearchStrategy = Field(default_factory=lambda: ToolSearchStrategyConfigSerializer().validate_dict({"tool_search_strategy_type": ToolSearchStrategyConfigSerializer.default_strategy})) + post_processing: List[ToolPostProcessor] = Field(default_factory=list) + manual_call_templates: List[CallTemplate] = Field(default_factory=list) + + @field_serializer("tool_repository") + def serialize_tool_repository(self, v: ConcurrentToolRepository): + return ConcurrentToolRepositoryConfigSerializer().to_dict(v) + + @field_validator("tool_repository", mode="before") + @classmethod + def validate_tool_repository(cls, v: Union[ConcurrentToolRepository, dict]): + if isinstance(v, ConcurrentToolRepository): + return v + return ConcurrentToolRepositoryConfigSerializer().validate_dict(v) + + @field_serializer("tool_search_strategy") + def serialize_tool_search_strategy(self, v: ToolSearchStrategy): + return ToolSearchStrategyConfigSerializer().to_dict(v) + + @field_validator("tool_search_strategy", mode="before") + @classmethod + def validate_tool_search_strategy(cls, v: Union[ToolSearchStrategy, dict]): + if isinstance(v, ToolSearchStrategy): + return v + return ToolSearchStrategyConfigSerializer().validate_dict(v) + + @field_serializer("load_variables_from") + def serialize_load_variables_from(self, v: Optional[List[VariableLoader]]): + if v is None: + return None + return [VariableLoaderSerializer().to_dict(item) for item in v] + + @field_validator("load_variables_from", mode="before") + @classmethod + def validate_load_variables_from(cls, v: Optional[List[Union[VariableLoader, dict]]]): + if v is None: + return None + return [item if isinstance(item, VariableLoader) else VariableLoaderSerializer().validate_dict(item) for item in v] + + @field_serializer("manual_call_templates") + def serialize_manual_call_templates(self, v: List[CallTemplate]): + return [CallTemplateSerializer().to_dict(v) for v in v] + + @field_validator("manual_call_templates", mode="before") + @classmethod + def validate_manual_call_templates(cls, v: List[Union[CallTemplate, dict]]): + return [v if isinstance(v, CallTemplate) else CallTemplateSerializer().validate_dict(v) for v in v] + + @field_serializer("post_processing") + def serialize_post_processing(self, v: List[ToolPostProcessor]): + return [ToolPostProcessorConfigSerializer().to_dict(v) for v in v] + + @field_validator("post_processing", mode="before") + @classmethod + def validate_post_processing(cls, v: List[Union[ToolPostProcessor, dict]]): + return [v if isinstance(v, ToolPostProcessor) else ToolPostProcessorConfigSerializer().validate_dict(v) for v in v] + +class UtcpClientConfigSerializer(Serializer[UtcpClientConfig]): + def to_dict(self, obj: UtcpClientConfig) -> dict: + return obj.model_dump() + + def validate_dict(self, data: dict) -> UtcpClientConfig: + try: + return UtcpClientConfig.model_validate(data) + except Exception as e: + raise UtcpSerializerValidationError("Invalid UtcpClientConfig: " + traceback.format_exc()) from e diff --git a/core/src/utcp/data/utcp_manual.py b/core/src/utcp/data/utcp_manual.py new file mode 100644 index 0000000..a7cfd67 --- /dev/null +++ b/core/src/utcp/data/utcp_manual.py @@ -0,0 +1,118 @@ +"""UTCP manual data structure for tool discovery. + +This module defines the UtcpManual model that standardizes the format for +tool provider responses during tool discovery. It serves as the contract +between tool providers and clients for sharing available tools and their +configurations. +""" + +from typing import List, Union, Optional, Any +from pydantic import BaseModel, field_serializer, field_validator +from utcp.python_specific_tooling.tool_decorator import ToolContext +from utcp.python_specific_tooling.version import __version__ +from utcp.data.tool import Tool +from utcp.data.tool import ToolSerializer +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +from utcp.plugins.plugin_loader import ensure_plugins_initialized +import traceback + +class UtcpManual(BaseModel): + """Standard format for tool provider responses during discovery. + + Represents the complete set of tools available from a provider, along + with version information for compatibility checking. This format is + returned by tool providers when clients query for available tools + (e.g., through the `/utcp` endpoint or similar discovery mechanisms). + + The manual serves as the authoritative source of truth for what tools + a provider offers and how they should be invoked. + + Attributes: + version: UTCP protocol version supported by the provider. + Defaults to the current library version. + tools: List of available tools with their complete configurations + including input/output schemas, descriptions, and metadata. + + Example: + ```python + @utcp_tool + def tool1(): + pass + + @utcp_tool + def tool2(): + pass + + # Create a manual from registered tools + manual = UtcpManual.create_from_decorators() + + # Manual with specific tools + manual = UtcpManual.create_from_decorators( + manual_version="1.0.0", + exclude=["tool1"] + ) + ``` + """ + utcp_version: str = __version__ + manual_version: str = "1.0.0" + tools: List[Tool] + + def __init__(self, tools: List[Tool], manual_version: str = "1.0.0", utcp_version: str = __version__): + super().__init__(utcp_version=utcp_version, manual_version=manual_version, tools=tools) + """Initializes the UtcpManual, ensuring plugins are loaded.""" + ensure_plugins_initialized() + + @staticmethod + def create_from_decorators(manual_version: str = "1.0.0", exclude: Optional[List[str]] = None) -> "UtcpManual": + """Create a UTCP manual from the global tool registry. + + Convenience method that creates a manual containing all tools + currently registered in the global ToolContext. This is typically + used by tool providers to generate their discovery response. + + Args: + version: UTCP protocol version to include in the manual. + Defaults to the current library version. + + Returns: + UtcpManual containing all registered tools and the specified version. + + Example: + ```python + # Create manual with default version + manual = UtcpManual.create_from_decorators() + + # Create manual with specific version + manual = UtcpManual.create_from_decorators(manual_version="1.2.0") + ``` + """ + if exclude is None: + exclude = [] + ensure_plugins_initialized() + return UtcpManual( + tools=[tool for tool in ToolContext.get_tools() if tool.name not in exclude], + manual_version=manual_version, + ) + + @field_serializer("tools") + def serialize_tools(self, tools: List[Tool]) -> List[dict]: + return [ToolSerializer().to_dict(tool) for tool in tools] + + @field_validator("tools", mode="before") + @classmethod + def validate_tools(cls, tools: List[Union[Tool, dict]]) -> List[Tool]: + return [v if isinstance(v, Tool) else ToolSerializer().validate_dict(v) for v in tools] + + +class UtcpManualSerializer(Serializer[UtcpManual]): + """Custom serializer for UtcpManual model.""" + + def to_dict(self, obj: UtcpManual) -> dict: + return obj.model_dump() + + def validate_dict(self, data: dict) -> UtcpManual: + try: + return UtcpManual.model_validate(data) + except Exception as e: + raise UtcpSerializerValidationError("Invalid UtcpManual: " + traceback.format_exc()) from e diff --git a/core/src/utcp/data/variable_loader.py b/core/src/utcp/data/variable_loader.py new file mode 100644 index 0000000..7dfdc3f --- /dev/null +++ b/core/src/utcp/data/variable_loader.py @@ -0,0 +1,47 @@ +from abc import ABC, abstractmethod +from dotenv import dotenv_values +from pydantic import BaseModel +from typing import Optional, Dict, Type +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +class VariableLoader(BaseModel, ABC): + """Abstract base class for variable loading configurations. + + Defines the interface for variable loaders that can retrieve variable + values from different sources such as files, databases, or external + services. Implementations provide specific loading mechanisms while + maintaining a consistent interface. + + Attributes: + variable_loader_type: Type identifier for the variable loader. + """ + variable_loader_type: str + + @abstractmethod + def get(self, key: str) -> Optional[str]: + """Retrieve a variable value by key. + + Args: + key: Variable name to retrieve. + + Returns: + Variable value if found, None otherwise. + """ + pass + +class VariableLoaderSerializer(Serializer[VariableLoader]): + """Custom serializer for VariableLoader model.""" + loader_serializers: Dict[str, Type[Serializer[VariableLoader]]] = {} + + def to_dict(self, obj: VariableLoader) -> dict: + return VariableLoaderSerializer.loader_serializers[obj.variable_loader_type].to_dict(obj) + + def validate_dict(self, data: dict) -> VariableLoader: + try: + return VariableLoaderSerializer.loader_serializers[data["variable_loader_type"]].validate_dict(data) + except KeyError: + raise ValueError(f"Invalid variable loader type: {data['variable_loader_type']}") + except Exception as e: + raise UtcpSerializerValidationError("Invalid VariableLoader: " + traceback.format_exc()) from e diff --git a/core/src/utcp/data/variable_loader_implementations/__init__.py b/core/src/utcp/data/variable_loader_implementations/__init__.py new file mode 100644 index 0000000..4396199 --- /dev/null +++ b/core/src/utcp/data/variable_loader_implementations/__init__.py @@ -0,0 +1,6 @@ +from utcp.data.variable_loader_implementations.dot_env_variable_loader import DotEnvVariableLoader, DotEnvVariableLoaderSerializer + +__all__ = [ + "DotEnvVariableLoader", + "DotEnvVariableLoaderSerializer", +] diff --git a/core/src/utcp/data/variable_loader_implementations/dot_env_variable_loader.py b/core/src/utcp/data/variable_loader_implementations/dot_env_variable_loader.py new file mode 100644 index 0000000..2cff413 --- /dev/null +++ b/core/src/utcp/data/variable_loader_implementations/dot_env_variable_loader.py @@ -0,0 +1,46 @@ +from utcp.data.variable_loader import VariableLoader +from typing import Optional, Literal +from dotenv import dotenv_values +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +class DotEnvVariableLoader(VariableLoader): + """Environment file variable loader implementation. + + Loads variables from .env files using the dotenv format. This loader + supports the standard key=value format with optional quoting and + comment support provided by the python-dotenv library. + + Attributes: + env_file_path: Path to the .env file to load variables from. + + Example: + ```python + loader = DotEnvVariableLoader(env_file_path=".env") + api_key = loader.get("API_KEY") + ``` + """ + variable_loader_type: Literal["dotenv"] = "dotenv" + env_file_path: str + + def get(self, key: str) -> Optional[str]: + """Load a variable from the configured .env file. + + Args: + key: Variable name to retrieve from the environment file. + + Returns: + Variable value if found in the file, None otherwise. + """ + return dotenv_values(self.env_file_path).get(key) + +class DotEnvVariableLoaderSerializer(Serializer[DotEnvVariableLoader]): + def to_dict(self, obj: DotEnvVariableLoader) -> dict: + return obj.model_dump() + + def validate_dict(self, data: dict) -> DotEnvVariableLoader: + try: + return DotEnvVariableLoader.model_validate(data) + except Exception as e: + raise UtcpSerializerValidationError("Invalid DotEnvVariableLoader: " + traceback.format_exc()) from e diff --git a/core/src/utcp/exceptions/__init__.py b/core/src/utcp/exceptions/__init__.py new file mode 100644 index 0000000..a33c4f6 --- /dev/null +++ b/core/src/utcp/exceptions/__init__.py @@ -0,0 +1,7 @@ +from utcp.exceptions.utcp_variable_not_found_exception import UtcpVariableNotFound +from utcp.exceptions.utcp_serializer_validation_error import UtcpSerializerValidationError + +__all__ = [ + "UtcpVariableNotFound", + "UtcpSerializerValidationError" +] diff --git a/core/src/utcp/exceptions/utcp_serializer_validation_error.py b/core/src/utcp/exceptions/utcp_serializer_validation_error.py new file mode 100644 index 0000000..c640408 --- /dev/null +++ b/core/src/utcp/exceptions/utcp_serializer_validation_error.py @@ -0,0 +1,2 @@ +class UtcpSerializerValidationError(Exception): + """Exception raised when a serializer validation fails.""" diff --git a/core/src/utcp/exceptions/utcp_variable_not_found_exception.py b/core/src/utcp/exceptions/utcp_variable_not_found_exception.py new file mode 100644 index 0000000..80fd2be --- /dev/null +++ b/core/src/utcp/exceptions/utcp_variable_not_found_exception.py @@ -0,0 +1,21 @@ +class UtcpVariableNotFound(Exception): + """Exception raised when a required variable cannot be found. + + This exception is thrown during variable substitution when a referenced + variable cannot be resolved through any of the configured variable sources. + It provides information about which variable was missing to help with + debugging configuration issues. + + Attributes: + variable_name: The name of the variable that could not be found. + """ + variable_name: str + + def __init__(self, variable_name: str): + """Initialize the exception with the missing variable name. + + Args: + variable_name: Name of the variable that could not be found. + """ + self.variable_name = variable_name + super().__init__(f"Variable {variable_name} referenced in provider configuration not found. Please add it to the environment variables or to your UTCP configuration.") diff --git a/core/src/utcp/implementations/__init__.py b/core/src/utcp/implementations/__init__.py new file mode 100644 index 0000000..12fb6d6 --- /dev/null +++ b/core/src/utcp/implementations/__init__.py @@ -0,0 +1,7 @@ +from utcp.implementations.in_mem_tool_repository import InMemToolRepository +from utcp.implementations.tag_search import TagAndDescriptionWordMatchStrategy + +__all__ = [ + "InMemToolRepository", + "TagAndDescriptionWordMatchStrategy", +] diff --git a/src/utcp/client/variable_substitutor.py b/core/src/utcp/implementations/default_variable_substitutor.py similarity index 66% rename from src/utcp/client/variable_substitutor.py rename to core/src/utcp/implementations/default_variable_substitutor.py index 1e69716..f82932a 100644 --- a/src/utcp/client/variable_substitutor.py +++ b/core/src/utcp/implementations/default_variable_substitutor.py @@ -10,55 +10,14 @@ Provider-specific variables are automatically namespaced to avoid conflicts. """ -from abc import ABC, abstractmethod -from utcp.client.utcp_client_config import UtcpClientConfig from typing import Any import os import re -from utcp.client.utcp_client_config import UtcpVariableNotFound +from utcp.exceptions import UtcpVariableNotFound from typing import List, Optional +from utcp.interfaces.variable_substitutor import VariableSubstitutor +from utcp.data.utcp_client_config import UtcpClientConfig -class VariableSubstitutor(ABC): - """Abstract interface for variable substitution implementations. - - Defines the contract for variable substitution systems that can replace - placeholders in configuration data with actual values from various sources. - Implementations handle different variable resolution strategies and - source hierarchies. - """ - - @abstractmethod - def substitute(self, obj: dict | list | str, config: UtcpClientConfig, provider_name: Optional[str] = None) -> Any: - """Substitute variables in the given object. - - Args: - obj: Object containing potential variable references to substitute. - Can be dict, list, str, or any other type. - config: UTCP client configuration containing variable definitions - and loaders. - provider_name: Optional provider name for variable namespacing. - - Returns: - Object with all variable references replaced by their values. - - Raises: - UtcpVariableNotFound: If a referenced variable cannot be resolved. - """ - pass - - @abstractmethod - def find_required_variables(self, obj: dict | list | str, provider_name: str) -> List[str]: - """Find all variable references in the given object. - - Args: - obj: Object to scan for variable references. - provider_name: Provider name for variable namespacing. - - Returns: - List of fully-qualified variable names found in the object. - """ - pass - class DefaultVariableSubstitutor(VariableSubstitutor): """Default implementation of variable substitution. @@ -80,7 +39,7 @@ class DefaultVariableSubstitutor(VariableSubstitutor): to avoid conflicts. For example, a variable 'api_key' for provider 'web_scraper' becomes 'web__scraper_api_key' internally. """ - def _get_variable(self, key: str, config: UtcpClientConfig, provider_name: Optional[str] = None) -> str: + def _get_variable(self, key: str, config: UtcpClientConfig, variable_namespace: Optional[str] = None) -> str: """Resolve a variable value through the hierarchical resolution system. Searches for the variable value in the following order: @@ -91,8 +50,8 @@ def _get_variable(self, key: str, config: UtcpClientConfig, provider_name: Optio Args: key: Variable name to resolve. config: UTCP client configuration containing variable sources. - provider_name: Optional provider name for variable namespacing. - When provided, the key is prefixed with the provider name. + variable_namespace: Optional variable namespace. + When provided, the key is prefixed with the variable namespace. Returns: Resolved variable value as a string. @@ -100,8 +59,8 @@ def _get_variable(self, key: str, config: UtcpClientConfig, provider_name: Optio Raises: UtcpVariableNotFound: If the variable cannot be found in any source. """ - if provider_name: - key = provider_name.replace("_", "!").replace("!", "__") + "_" + key + if variable_namespace: + key = variable_namespace.replace("_", "!").replace("!", "__") + "_" + key if config.variables and key in config.variables: return config.variables[key] if config.load_variables_from: @@ -118,7 +77,7 @@ def _get_variable(self, key: str, config: UtcpClientConfig, provider_name: Optio raise UtcpVariableNotFound(key) - def substitute(self, obj: dict | list | str, config: UtcpClientConfig, provider_name: Optional[str] = None) -> Any: + def substitute(self, obj: dict | list | str, config: UtcpClientConfig, variable_namespace: Optional[str] = None) -> Any: """Recursively substitute variables in nested data structures. Performs deep substitution on dictionaries, lists, and strings. @@ -128,7 +87,7 @@ def substitute(self, obj: dict | list | str, config: UtcpClientConfig, provider_ Args: obj: Object to perform substitution on. Can be any type. config: UTCP client configuration containing variable sources. - provider_name: Optional provider name for variable namespacing. + variable_namespace: Optional variable namespace. Returns: Object with all variable references replaced. Structure and @@ -136,6 +95,7 @@ def substitute(self, obj: dict | list | str, config: UtcpClientConfig, provider_ Raises: UtcpVariableNotFound: If any referenced variable cannot be resolved. + ValueError: If variable_namespace contains invalid characters. Example: ```python @@ -148,36 +108,42 @@ def substitute(self, obj: dict | list | str, config: UtcpClientConfig, provider_ # Returns: {"url": "https://api.example.com/api", "port": 8080} ``` """ + # Check that variable_namespace only contains alphanumeric characters or underscores + if variable_namespace and not all(c.isalnum() or c == '_' for c in variable_namespace): + raise ValueError(f"Variable namespace '{variable_namespace}' contains invalid characters. Only alphanumeric characters and underscores are allowed.") + if isinstance(obj, dict): - return {k: self.substitute(v, config, provider_name) for k, v in obj.items()} + return {k: self.substitute(v, config, variable_namespace) for k, v in obj.items()} elif isinstance(obj, list): - return [self.substitute(elem, config, provider_name) for elem in obj] + return [self.substitute(elem, config, variable_namespace) for elem in obj] elif isinstance(obj, str): # Use a regular expression to find all variables in the string, supporting ${VAR} and $VAR formats def replacer(match): # The first group that is not None is the one that matched var_name = next((g for g in match.groups() if g is not None), "") - return self._get_variable(var_name, config, provider_name) + return self._get_variable(var_name, config, variable_namespace) return re.sub(r'\${(\w+)}|\$(\w+)', replacer, obj) else: return obj - def find_required_variables(self, obj: dict | list | str, provider_name: str) -> List[str]: + def find_required_variables(self, obj: dict | list | str, variable_namespace: Optional[str] = None) -> List[str]: """Recursively discover all variable references in a data structure. Scans the object for variable references using ${VAR} and $VAR syntax, - returning fully-qualified variable names with provider namespacing. + returning fully-qualified variable names with variable namespacing. Useful for validation and dependency analysis. Args: obj: Object to scan for variable references. - provider_name: Provider name used for variable namespacing. - Variable names are prefixed with this provider name. + variable_namespace: Variable namespace used for variable namespacing. + Variable names are prefixed with this variable namespace. + + Raises: + ValueError: If variable_namespace contains invalid characters. Returns: List of fully-qualified variable names found in the object. - Variables are prefixed with the provider name to avoid conflicts. Example: ```python @@ -189,16 +155,20 @@ def find_required_variables(self, obj: dict | list | str, provider_name: str) -> # Returns: ["web__api_HOST", "web__api_API_KEY"] ``` """ + # Check that variable_namespace only contains alphanumeric characters or underscores + if variable_namespace and not all(c.isalnum() or c == '_' for c in variable_namespace): + raise ValueError(f"Variable namespace '{variable_namespace}' contains invalid characters. Only alphanumeric characters and underscores are allowed.") + if isinstance(obj, dict): result = [] for v in obj.values(): - vars = self.find_required_variables(v, provider_name) + vars = self.find_required_variables(v, variable_namespace) result.extend(vars) return result elif isinstance(obj, list): result = [] for elem in obj: - vars = self.find_required_variables(elem, provider_name) + vars = self.find_required_variables(elem, variable_namespace) result.extend(vars) return result elif isinstance(obj, str): @@ -209,7 +179,10 @@ def find_required_variables(self, obj: dict | list | str, provider_name: str) -> for match in re.finditer(pattern, obj): # The first group that is not None is the one that matched var_name = next(g for g in match.groups() if g is not None) - full_var_name = provider_name.replace("_", "!").replace("!", "__") + "_" + var_name + if variable_namespace: + full_var_name = variable_namespace.replace("_", "!").replace("!", "__") + "_" + var_name + else: + full_var_name = var_name variables.append(full_var_name) return variables diff --git a/core/src/utcp/implementations/in_mem_tool_repository.py b/core/src/utcp/implementations/in_mem_tool_repository.py new file mode 100644 index 0000000..0a6efbc --- /dev/null +++ b/core/src/utcp/implementations/in_mem_tool_repository.py @@ -0,0 +1,113 @@ +from typing import List, Dict, Optional + +from utcp.data.utcp_manual import UtcpManual +from utcp.python_specific_tooling.async_rwlock import AsyncRWLock +from utcp.data.call_template import CallTemplate +from utcp.data.tool import Tool +from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepository +from utcp.interfaces.serializer import Serializer + +class InMemToolRepository(ConcurrentToolRepository): + """Thread-safe in-memory implementation of `ConcurrentToolRepository`. + + Stores tools and their associated manual call templates in dictionaries and + protects all operations with a read-write lock to ensure consistency under + concurrency while allowing multiple concurrent readers. + """ + + def __init__(self): + super().__init__(tool_repository_type="in_memory") + # RW lock to allow concurrent reads and exclusive writes + self._rwlock = AsyncRWLock() + + # Tool name -> Tool + self._tools_by_name: Dict[str, Tool] = {} + + # Manual name -> UtcpManual + self._manuals: Dict[str, UtcpManual] = {} + + # Manual name -> CallTemplate + self._manual_call_templates: Dict[str, CallTemplate] = {} + + async def save_manual(self, manual_call_template: CallTemplate, manual: UtcpManual) -> None: + async with self._rwlock.write(): + manual_name = manual_call_template.name + + # Remove old tools for this manual from the global index + old_manual = self._manuals.get(manual_name) + if old_manual is not None: + for t in old_manual.tools: + self._tools_by_name.pop(t.name, None) + + # Save/replace manual and its tools + self._manual_call_templates[manual_name] = manual_call_template + self._manuals[manual_name] = manual + + # Index tools globally by name + for t in manual.tools: + self._tools_by_name[t.name] = t + + async def remove_manual(self, manual_name: str) -> bool: + async with self._rwlock.write(): + # Remove tools of this manual + old_manual = self._manuals.get(manual_name) + if old_manual is not None: + for t in old_manual.tools: + self._tools_by_name.pop(t.name, None) + else: + return False + + # Remove manual and mapping + self._manuals.pop(manual_name, None) + self._manual_call_templates.pop(manual_name, None) + return True + + async def remove_tool(self, tool_name: str) -> bool: + async with self._rwlock.write(): + tool = self._tools_by_name.pop(tool_name, None) + if tool is None: + return False + + # Remove from any manual lists + for manual in self._manuals.values(): + if tool in manual.tools: + manual.tools.remove(tool) + return True + + async def get_tool(self, tool_name: str) -> Optional[Tool]: + async with self._rwlock.read(): + return self._tools_by_name.get(tool_name) + + async def get_tools(self) -> List[Tool]: + async with self._rwlock.read(): + return list(self._tools_by_name.values()) + + async def get_tools_by_manual(self, manual_name: str) -> Optional[List[Tool]]: + async with self._rwlock.read(): + manual = self._manuals.get(manual_name) + return manual.tools if manual is not None else None + + async def get_manual(self, manual_name: str) -> Optional[UtcpManual]: + async with self._rwlock.read(): + return self._manuals.get(manual_name) + + async def get_manuals(self) -> List[UtcpManual]: + async with self._rwlock.read(): + return list(self._manuals.values()) + + async def get_manual_call_template(self, manual_call_template_name: str) -> Optional[CallTemplate]: + async with self._rwlock.read(): + return self._manual_call_templates.get(manual_call_template_name) + + async def get_manual_call_templates(self) -> List[CallTemplate]: + async with self._rwlock.read(): + return list(self._manual_call_templates.values()) + +class InMemToolRepositoryConfigSerializer(Serializer[InMemToolRepository]): + def to_dict(self, obj: InMemToolRepository) -> dict: + return { + "tool_repository_type": obj.tool_repository_type, + } + + def validate_dict(self, data: dict) -> InMemToolRepository: + return InMemToolRepository() diff --git a/core/src/utcp/implementations/post_processors/__init__.py b/core/src/utcp/implementations/post_processors/__init__.py new file mode 100644 index 0000000..67d829c --- /dev/null +++ b/core/src/utcp/implementations/post_processors/__init__.py @@ -0,0 +1,9 @@ +from utcp.implementations.post_processors.filter_dict_post_processor import FilterDictPostProcessor, FilterDictPostProcessorConfigSerializer +from utcp.implementations.post_processors.limit_strings_post_processor import LimitStringsPostProcessor, LimitStringsPostProcessorConfigSerializer + +__all__ = [ + "FilterDictPostProcessor", + "FilterDictPostProcessorConfigSerializer", + "LimitStringsPostProcessor", + "LimitStringsPostProcessorConfigSerializer", +] diff --git a/core/src/utcp/implementations/post_processors/filter_dict_post_processor.py b/core/src/utcp/implementations/post_processors/filter_dict_post_processor.py new file mode 100644 index 0000000..fe6d082 --- /dev/null +++ b/core/src/utcp/implementations/post_processors/filter_dict_post_processor.py @@ -0,0 +1,99 @@ +from utcp.interfaces.tool_post_processor import ToolPostProcessor +from utcp.data.tool import Tool +from utcp.data.call_template import CallTemplate +from typing import Any, List, Optional, TYPE_CHECKING, Literal +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +if TYPE_CHECKING: + from utcp.utcp_client import UtcpClient + +class FilterDictPostProcessor(ToolPostProcessor): + tool_post_processor_type: Literal["filter_dict"] = "filter_dict" + exclude_keys: Optional[List[str]] = None + only_include_keys: Optional[List[str]] = None + exclude_tools: Optional[List[str]] = None + only_include_tools: Optional[List[str]] = None + exclude_manuals: Optional[List[str]] = None + only_include_manuals: Optional[List[str]] = None + + def post_process(self, caller: 'UtcpClient', tool: Tool, manual_call_template: 'CallTemplate', result: Any) -> Any: + if self.exclude_tools and tool.name in self.exclude_tools: + return result + if self.only_include_tools and tool.name not in self.only_include_tools: + return result + if self.exclude_manuals and manual_call_template.name in self.exclude_manuals: + return result + if self.only_include_manuals and manual_call_template.name not in self.only_include_manuals: + return result + + if not self.exclude_keys and not self.only_include_keys: + return result + if self.exclude_keys: + result = self._filter_dict_exclude_keys(result) + if self.only_include_keys: + result = self._filter_dict_only_include_keys(result) + return result + + def _filter_dict_exclude_keys(self, result: Any) -> Any: + if isinstance(result, dict): + new_result = {} + for key, value in result.items(): + if key not in self.exclude_keys: + new_result[key] = self._filter_dict(value) + return new_result + + if isinstance(result, list): + new_list = [] + for item in result: + processed_item = self._filter_dict_exclude_keys(item) + if isinstance(processed_item, dict): + if processed_item: + new_list.append(processed_item) + elif isinstance(processed_item, list): + if processed_item: + new_list.append(processed_item) + else: + new_list.append(processed_item) + return new_list + + return result + + def _filter_dict_only_include_keys(self, result: Any) -> Any: + if isinstance(result, dict): + new_result = {} + for key, value in result.items(): + if key in self.only_include_keys: + if isinstance(value, dict): + new_result[key] = self._filter_dict_only_include_keys(value) + else: + new_result[key] = value + else: + processed_value = self._filter_dict_only_include_keys(value) + if (isinstance(processed_value, dict) and processed_value) or \ + (isinstance(processed_value, list) and processed_value): + new_result[key] = processed_value + return new_result + + if isinstance(result, list): + new_list = [] + for item in result: + processed_item = self._filter_dict_only_include_keys(item) + if isinstance(processed_item, dict) and processed_item: + new_list.append(processed_item) + if isinstance(processed_item, list) and processed_item: + new_list.append(processed_item) + return new_list + + return result + +class FilterDictPostProcessorConfigSerializer(Serializer[FilterDictPostProcessor]): + def to_dict(self, obj: FilterDictPostProcessor) -> dict: + return obj.model_dump() + + def validate_dict(self, data: dict) -> FilterDictPostProcessor: + try: + return FilterDictPostProcessor.model_validate(data) + except Exception as e: + raise UtcpSerializerValidationError("Invalid FilterDictPostProcessor: " + traceback.format_exc()) from e diff --git a/core/src/utcp/implementations/post_processors/limit_strings_post_processor.py b/core/src/utcp/implementations/post_processors/limit_strings_post_processor.py new file mode 100644 index 0000000..c0a19a1 --- /dev/null +++ b/core/src/utcp/implementations/post_processors/limit_strings_post_processor.py @@ -0,0 +1,49 @@ +from utcp.interfaces.tool_post_processor import ToolPostProcessor +from utcp.data.tool import Tool +from utcp.data.call_template import CallTemplate +from typing import Any, List, Optional, TYPE_CHECKING, Literal +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +if TYPE_CHECKING: + from utcp.utcp_client import UtcpClient + +class LimitStringsPostProcessor(ToolPostProcessor): + tool_post_processor_type: Literal["limit_strings"] = "limit_strings" + limit: int = 10000 + exclude_tools: Optional[List[str]] = None + only_include_tools: Optional[List[str]] = None + exclude_manuals: Optional[List[str]] = None + only_include_manuals: Optional[List[str]] = None + + def post_process(self, caller: 'UtcpClient', tool: Tool, manual_call_template: 'CallTemplate', result: Any) -> Any: + if self.exclude_tools and tool.name in self.exclude_tools: + return result + if self.only_include_tools and tool.name not in self.only_include_tools: + return result + if self.exclude_manuals and manual_call_template.name in self.exclude_manuals: + return result + if self.only_include_manuals and manual_call_template.name not in self.only_include_manuals: + return result + + return self._process_object(result) + + def _process_object(self, obj: Any) -> Any: + if isinstance(obj, str): + return obj[:self.limit] + if isinstance(obj, list): + return [self._process_object(item) for item in obj] + if isinstance(obj, dict): + return {key: self._process_object(value) for key, value in obj.items()} + return obj + +class LimitStringsPostProcessorConfigSerializer(Serializer[LimitStringsPostProcessor]): + def to_dict(self, obj: LimitStringsPostProcessor) -> dict: + return obj.model_dump() + + def validate_dict(self, data: dict) -> LimitStringsPostProcessor: + try: + return LimitStringsPostProcessor.model_validate(data) + except Exception as e: + raise UtcpSerializerValidationError("Invalid LimitStringsPostProcessor: " + traceback.format_exc()) from e diff --git a/src/utcp/client/tool_search_strategies/tag_search.py b/core/src/utcp/implementations/tag_search.py similarity index 66% rename from src/utcp/client/tool_search_strategies/tag_search.py rename to core/src/utcp/implementations/tag_search.py index 57fcd37..b35e9db 100644 --- a/src/utcp/client/tool_search_strategies/tag_search.py +++ b/core/src/utcp/implementations/tag_search.py @@ -5,15 +5,15 @@ explicit tag matches receive higher scores than description word matches. """ -from utcp.client.tool_search_strategy import ToolSearchStrategy -from typing import List, Dict, Tuple -from utcp.shared.tool import Tool -from utcp.client.tool_repository import ToolRepository +from utcp.interfaces.tool_search_strategy import ToolSearchStrategy +from typing import List, Tuple, Optional, Literal +from utcp.data.tool import Tool +from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepository import re -import asyncio +from utcp.interfaces.serializer import Serializer -class TagSearchStrategy(ToolSearchStrategy): - """Tag-based search strategy for UTCP tools. +class TagAndDescriptionWordMatchStrategy(ToolSearchStrategy): + """Tag and description word match search strategy for UTCP tools. Implements a weighted scoring algorithm that matches search queries against tool tags and descriptions. Explicit tag matches receive full weight while @@ -26,35 +26,18 @@ class TagSearchStrategy(ToolSearchStrategy): - Only considers description words longer than 2 characters Examples: - >>> strategy = TagSearchStrategy(repository, description_weight=0.3) + >>> strategy = TagAndDescriptionWordMatchStrategy(description_weight=0.3) >>> tools = await strategy.search_tools("weather api", limit=5) >>> # Returns tools with "weather" or "api" tags/descriptions Attributes: - tool_repository: Repository to search for tools. description_weight: Weight multiplier for description matches (0.0-1.0). """ + tool_search_strategy_type: Literal["tag_and_description_word_match"] = "tag_and_description_word_match" + description_weight: float = 1 + tag_weight: float = 3 - def __init__(self, tool_repository: ToolRepository, description_weight: float = 0.3): - """Initialize the tag search strategy. - - Args: - tool_repository: Repository containing tools to search. - description_weight: Weight for description word matches relative to - tag matches. Should be between 0.0 and 1.0, where 1.0 gives - equal weight to tags and descriptions. - - Raises: - ValueError: If description_weight is not between 0.0 and 1.0. - """ - if not 0.0 <= description_weight <= 1.0: - raise ValueError("description_weight must be between 0.0 and 1.0") - - self.tool_repository = tool_repository - # Weight for description words vs explicit tags (explicit tags have weight of 1.0) - self.description_weight = description_weight - - async def search_tools(self, query: str, limit: int = 10) -> List[Tool]: + async def search_tools(self, tool_repository: ConcurrentToolRepository, query: str, limit: int = 10, any_of_tags_required: Optional[List[str]] = None) -> List[Tool]: """Search tools using tag and description matching. Implements a weighted scoring system that ranks tools based on how well @@ -70,6 +53,8 @@ async def search_tools(self, query: str, limit: int = 10) -> List[Tool]: Args: query: Search query string. Case-insensitive, word-based matching. limit: Maximum number of tools to return. Must be >= 0. + any_of_tags_required: Optional list of tags where one of them must be present in the tool's tags + for it to be considered a match. Returns: List of Tool objects ranked by relevance score (highest first). @@ -85,8 +70,11 @@ async def search_tools(self, query: str, limit: int = 10) -> List[Tool]: # Extract words from the query, filtering out non-word characters query_words = set(re.findall(r'\w+', query_lower)) - # Get all tools (using asyncio to run the coroutine) - tools = await self.tool_repository.get_tools() + # Get all tools + tools: List[Tool] = await tool_repository.get_tools() + + if any_of_tags_required is not None and len(any_of_tags_required) > 0: + tools = [tool for tool in tools if any(tag in tool.tags for tag in any_of_tags_required)] # Calculate scores for each tool tool_scores: List[Tuple[Tool, float]] = [] @@ -99,12 +87,14 @@ async def search_tools(self, query: str, limit: int = 10) -> List[Tool]: tag_lower = tag.lower() # Check if the tag appears in the query if tag_lower in query_lower: - score += 1.0 + score += self.tag_weight + continue # Also check if the tag words match query words tag_words = set(re.findall(r'\w+', tag_lower)) for word in tag_words: if word in query_words: - score += self.description_weight # Partial match for tag words + score += self.tag_weight + break # Score from description (with lower weight) if tool.description: @@ -120,3 +110,13 @@ async def search_tools(self, query: str, limit: int = 10) -> List[Tool]: # Return up to 'limit' tools return sorted_tools[:limit] + +class TagAndDescriptionWordMatchStrategyConfigSerializer(Serializer[TagAndDescriptionWordMatchStrategy]): + def to_dict(self, obj: TagAndDescriptionWordMatchStrategy) -> dict: + return obj.model_dump() + + def validate_dict(self, data: dict) -> TagAndDescriptionWordMatchStrategy: + try: + return TagAndDescriptionWordMatchStrategy.model_validate(data) + except Exception as e: + raise ValueError(f"Invalid configuration: {e}") from e diff --git a/core/src/utcp/implementations/utcp_client_implementation.py b/core/src/utcp/implementations/utcp_client_implementation.py new file mode 100644 index 0000000..209904a --- /dev/null +++ b/core/src/utcp/implementations/utcp_client_implementation.py @@ -0,0 +1,193 @@ +from utcp.data.utcp_manual import UtcpManual + +import re +import os +import json +import asyncio +from typing import Dict, Any, List, Union, Optional, AsyncGenerator, TYPE_CHECKING + +from utcp.data.call_template import CallTemplate +from utcp.data.call_template import CallTemplateSerializer +from utcp.data.tool import Tool +from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepositoryConfigSerializer, ConcurrentToolRepository +from utcp.interfaces.tool_search_strategy import ToolSearchStrategyConfigSerializer, ToolSearchStrategy +from utcp.interfaces.variable_substitutor import VariableSubstitutor +from utcp.data.utcp_client_config import UtcpClientConfig, UtcpClientConfigSerializer +from utcp.implementations.default_variable_substitutor import DefaultVariableSubstitutor +from utcp.implementations.tag_search import TagAndDescriptionWordMatchStrategy +from utcp.exceptions import UtcpVariableNotFound +from utcp.data.register_manual_response import RegisterManualResult +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.exceptions import UtcpSerializerValidationError +import traceback +from utcp.utcp_client import UtcpClient +import logging + +logger = logging.getLogger(__name__) + +class UtcpClientImplementation(UtcpClient): + def __init__( + self, + config: UtcpClientConfig, + variable_substitutor: VariableSubstitutor, + root_dir: str, + ): + super().__init__(config, root_dir) + self.variable_substitutor = variable_substitutor + + @classmethod + async def create( + cls, + root_dir: Optional[str] = None, + config: Optional[Union[str, Dict[str, Any], UtcpClientConfig]] = None, + ) -> 'UtcpClient': + # Validate and load the config + client_config_serializer = UtcpClientConfigSerializer() + if config is None: + config = UtcpClientConfig() + elif isinstance(config, dict): + config = client_config_serializer.validate_dict(config) + elif isinstance(config, str): + try: + with open(config, "r") as f: + file_content = f.read() + config = client_config_serializer.validate_dict(json.loads(file_content)) + except UtcpSerializerValidationError as e: + raise e + except Exception as e: + raise ValueError(f"Invalid config file: {config}, error: {traceback.format_exc()}") from e + + # Set the root directory + if root_dir is None: + root_dir = os.getcwd() + + # Create the client + client = cls(config, DefaultVariableSubstitutor(), root_dir) + + # Substitute variables in the config + if client.config.variables: + config_without_vars = client_config_serializer.copy(client.config) + config_without_vars.variables = None + client.config.variables = client.variable_substitutor.substitute(client.config.variables, config_without_vars) + + # Load the manuals if any + if config.manual_call_templates: + await client.register_manuals(config.manual_call_templates) + + return client + + async def register_manual(self, manual_call_template: CallTemplate) -> RegisterManualResult: + # Replace all non-word characters with underscore + manual_call_template.name = re.sub(r'[^\w]', '_', manual_call_template.name) + if await self.config.tool_repository.get_manual(manual_call_template.name) is not None: + raise ValueError(f"Manual {manual_call_template.name} already registered, please use a different name or deregister the existing manual") + manual_call_template = self._substitute_call_template_variables(manual_call_template, manual_call_template.name) + if manual_call_template.call_template_type not in CommunicationProtocol.communication_protocols: + raise ValueError(f"No registered communication protocol of type {manual_call_template.call_template_type} found, available types: {CommunicationProtocol.communication_protocols.keys()}") + + result = await CommunicationProtocol.communication_protocols[manual_call_template.call_template_type].register_manual(self, manual_call_template) + + if result.success: + for tool in result.manual.tools: + if not tool.name.startswith(manual_call_template.name + "."): + tool.name = manual_call_template.name + "." + tool.name + await self.config.tool_repository.save_manual(result.manual_call_template, result.manual) + + return result + + async def register_manuals(self, manual_call_templates: List[CallTemplate]) -> List[RegisterManualResult]: + # Create tasks for parallel CallTemplate registration + tasks = [] + for manual_call_template in manual_call_templates: + async def try_register_manual(manual_call_template=manual_call_template): + try: + result = await self.register_manual(manual_call_template) + if result.success: + logger.info(f"Successfully registered manual '{manual_call_template.name}' with {len(result.manual.tools)} tools") + else: + logger.error(f"Error registering manual '{manual_call_template.name}': {result.errors}") + return result + except UtcpVariableNotFound as e: + raise e + except Exception as e: + logger.error(f"Error registering manual '{manual_call_template.name}': {traceback.format_exc()}") + return RegisterManualResult( + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + success=False, + errors=[traceback.format_exc()] + ) + + tasks.append(try_register_manual()) + + # Wait for all tasks to complete and collect results + results = await asyncio.gather(*tasks) + return [p for p in results if p is not None] + + async def deregister_manual(self, manual_name: str) -> bool: + manual_call_template = await self.config.tool_repository.get_manual_call_template(manual_name) + if manual_call_template is None: + return False + await CommunicationProtocol.communication_protocols[manual_call_template.call_template_type].deregister_manual(self, manual_call_template) + return await self.config.tool_repository.remove_manual(manual_name) + + async def call_tool(self, tool_name: str, tool_args: Dict[str, Any]) -> Any: + manual_name = tool_name.split(".")[0] + tool = await self.config.tool_repository.get_tool(tool_name) + if tool is None: + raise ValueError(f"Tool not found: {tool_name}") + tool_call_template = tool.tool_call_template + tool_call_template = self._substitute_call_template_variables(tool_call_template, manual_name) + result = await CommunicationProtocol.communication_protocols[tool_call_template.call_template_type].call_tool(self, tool_name, tool_args, tool_call_template) + + for post_processor in self.config.post_processing: + result = post_processor.post_process(self, tool, tool_call_template, result) + return result + + async def call_tool_streaming(self, tool_name: str, tool_args: Dict[str, Any]) -> AsyncGenerator[Any, None]: + manual_name = tool_name.split(".")[0] + tool = await self.config.tool_repository.get_tool(tool_name) + if tool is None: + raise ValueError(f"Tool not found: {tool_name}") + tool_call_template = tool.tool_call_template + tool_call_template = self._substitute_call_template_variables(tool_call_template, manual_name) + async for item in CommunicationProtocol.communication_protocols[tool_call_template.call_template_type].call_tool_streaming(self, tool_name, tool_args, tool_call_template): + for post_processor in self.config.post_processing: + item = post_processor.post_process(self, tool, tool_call_template, item) + yield item + + async def search_tools(self, query: str, limit: int = 10, any_of_tags_required: Optional[List[str]] = None) -> List[Tool]: + return await self.config.tool_search_strategy.search_tools( + tool_repository=self.config.tool_repository, + query=query, + limit=limit, + any_of_tags_required=any_of_tags_required, + ) + + async def get_required_variables_for_manual_and_tools(self, manual_call_template: CallTemplate) -> List[str]: + manual_call_template.name = re.sub(r'[^\w]', '_', manual_call_template.name) + variables_for_CallTemplate = self.variable_substitutor.find_required_variables(CallTemplateSerializer().to_dict(manual_call_template), manual_call_template.name) + if len(variables_for_CallTemplate) > 0: + try: + manual_call_template = self._substitute_call_template_variables(manual_call_template, manual_call_template.name) + except UtcpVariableNotFound as e: + return variables_for_CallTemplate + return variables_for_CallTemplate + if manual_call_template.call_template_type not in CommunicationProtocol.communication_protocols: + raise ValueError(f"CallTemplate type not supported: {manual_call_template.call_template_type}") + register_manual_result: RegisterManualResult = await CommunicationProtocol.communication_protocols[manual_call_template.call_template_type].register_manual(self, manual_call_template) + for tool in register_manual_result.manual.tools: + variables_for_CallTemplate.extend(self.variable_substitutor.find_required_variables(CallTemplateSerializer().to_dict(tool.tool_call_template), manual_call_template.name)) + return variables_for_CallTemplate + + async def get_required_variables_for_registered_tool(self, tool_name: str) -> List[str]: + manual_name = tool_name.split(".")[0] + tool = await self.config.tool_repository.get_tool(tool_name) + if tool is None: + raise ValueError(f"Tool not found: {tool_name}") + return self.variable_substitutor.find_required_variables(CallTemplateSerializer().to_dict(tool.tool_call_template), manual_name) + + def _substitute_call_template_variables(self, call_template: CallTemplate, namespace: Optional[str] = None) -> CallTemplate: + call_template_dict = CallTemplateSerializer().to_dict(call_template) + processed_dict = self.variable_substitutor.substitute(call_template_dict, self.config, namespace) + return CallTemplateSerializer().validate_dict(processed_dict) diff --git a/src/utcp/shared/__init__.py b/core/src/utcp/interfaces/__init__.py similarity index 100% rename from src/utcp/shared/__init__.py rename to core/src/utcp/interfaces/__init__.py diff --git a/core/src/utcp/interfaces/communication_protocol.py b/core/src/utcp/interfaces/communication_protocol.py new file mode 100644 index 0000000..b3c31a1 --- /dev/null +++ b/core/src/utcp/interfaces/communication_protocol.py @@ -0,0 +1,115 @@ +"""Abstract interface for UTCP client transport implementations. + +This module defines the contract that all transport implementations must follow +to integrate with the UTCP client. Transport implementations handle the actual +communication with different types of tool providers (HTTP, CLI, WebSocket, etc.). +""" + +from abc import ABC, abstractmethod +from typing import Dict, Any, AsyncGenerator, TYPE_CHECKING +from utcp.data.register_manual_response import RegisterManualResult +from utcp.data.call_template import CallTemplate +if TYPE_CHECKING: + from utcp.utcp_client import UtcpClient + +class CommunicationProtocol(ABC): + """Abstract interface for UTCP client transport implementations. + + Defines the contract that all transport implementations must follow to + integrate with the UTCP client. Each transport handles communication + with a specific type of provider (HTTP, CLI, WebSocket, etc.). + + Transport implementations are responsible for: + - Discovering available tools from providers + - Managing provider lifecycle (registration/deregistration) + - Executing tool calls through the appropriate protocol + """ + communication_protocols: dict[str, 'CommunicationProtocol'] = {} + + @abstractmethod + async def register_manual(self, caller: 'UtcpClient', manual_call_template: CallTemplate) -> RegisterManualResult: + """Register a manual and its tools. + + Connects to the provider and retrieves the list of tools it offers. + This may involve making discovery requests, parsing configuration files, + or initializing connections depending on the provider type. + + Args: + caller: The UTCP client that is calling this method. + manual_call_template: The call template of the manual to register. + + Returns: + RegisterManualResult object containing the call template and manual. + + Raises: + ConnectionError: If unable to connect to the provider. + ValueError: If the provider configuration is invalid. + """ + pass + + @abstractmethod + async def deregister_manual(self, caller: 'UtcpClient', manual_call_template: CallTemplate) -> None: + """Deregister a manual and its tools. + + Cleanly disconnects from the provider and releases any associated + resources such as connections, processes, or file handles. + + Args: + caller: The UTCP client that is calling this method. + manual_call_template: The call template of the manual to deregister. + + Note: + Should handle cases where the provider is already disconnected + or was never properly registered. + """ + pass + + @abstractmethod + async def call_tool(self, caller: 'UtcpClient', tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + """Execute a tool call through this transport. + + Sends a tool invocation request to the provider using the appropriate + protocol and returns the result. Handles serialization of arguments + and deserialization of responses according to the transport type. + + Args: + caller: The UTCP client that is calling this method. + tool_name: Name of the tool to call (may include provider prefix). + tool_args: Dictionary of arguments to pass to the tool. + tool_call_template: Call template of the tool to call. + + Returns: + The tool's response, with type depending on the tool's output schema. + + Raises: + ToolNotFoundError: If the specified tool doesn't exist. + ValidationError: If the arguments don't match the tool's input schema. + ConnectionError: If unable to communicate with the provider. + TimeoutError: If the tool call exceeds the configured timeout. + """ + pass + + @abstractmethod + async def call_tool_streaming(self, caller: 'UtcpClient', tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> AsyncGenerator[Any, None]: + """Execute a tool call through this transport streamingly. + + Sends a tool invocation request to the provider using the appropriate + protocol and returns the result. Handles serialization of arguments + and deserialization of responses according to the transport type. + + Args: + caller: The UTCP client that is calling this method. + tool_name: Name of the tool to call (may include provider prefix). + tool_args: Dictionary of arguments to pass to the tool. + tool_call_template: Call template of the tool to call. + + Returns: + An async generator that yields the tool's response, with type depending on the tool's output schema. + + Raises: + ToolNotFoundError: If the specified tool doesn't exist. + ValidationError: If the arguments don't match the tool's input schema. + ConnectionError: If unable to communicate with the provider. + TimeoutError: If the tool call exceeds the configured timeout. + """ + pass diff --git a/core/src/utcp/interfaces/concurrent_tool_repository.py b/core/src/utcp/interfaces/concurrent_tool_repository.py new file mode 100644 index 0000000..fea47f2 --- /dev/null +++ b/core/src/utcp/interfaces/concurrent_tool_repository.py @@ -0,0 +1,173 @@ +"""Abstract interface for tool and provider storage. + +This module defines the contract for implementing tool repositories that store +and manage UTCP tools and their associated providers. Different implementations +can provide various storage backends such as in-memory, database, or file-based +storage. +""" + +from abc import ABC, abstractmethod +from typing import List, Dict, Any, Optional + +from utcp.data.call_template import CallTemplate +from utcp.data.tool import Tool +from utcp.data.utcp_manual import UtcpManual +from utcp.interfaces.serializer import Serializer +from pydantic import BaseModel +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +class ConcurrentToolRepository(BaseModel, ABC): + """Abstract interface for tool and provider storage implementations. + + Defines the contract for repositories that manage the lifecycle and storage + of UTCP tools and call templates. Repositories are responsible for: + - Persisting provider configurations and their associated tools + - Providing efficient lookup and retrieval operations + - Managing relationships between call templates and tools + - Ensuring data consistency during operations + - Thread safety + + The repository interface supports both individual and bulk operations, + allowing for flexible implementation strategies ranging from simple + in-memory storage to sophisticated database backends. + + Note: + All methods are async to support both synchronous and asynchronous + storage implementations. + """ + tool_repository_type: str + + @abstractmethod + async def save_manual(self, manual_call_template: CallTemplate, manual: UtcpManual) -> None: + """ + Save a manual and its tools in the repository. + + Args: + manual_call_template: The call template associated with the manual to save. + manual: The manual to save. + """ + pass + + @abstractmethod + async def remove_manual(self, manual_name: str) -> bool: + """ + Remove a manual and its tools from the repository. + + Args: + manual_name: The name of the manual to remove. + + Returns: + True if the manual was removed, False otherwise. + """ + pass + + @abstractmethod + async def remove_tool(self, tool_name: str) -> bool: + """ + Remove a tool from the repository. + + Args: + tool_name: The name of the tool to remove. + + Returns: + True if the tool was removed, False otherwise. + """ + pass + + @abstractmethod + async def get_tool(self, tool_name: str) -> Optional[Tool]: + """ + Get a tool from the repository. + + Args: + tool_name: The name of the tool to retrieve. + + Returns: + The tool if found, otherwise None. + """ + pass + + @abstractmethod + async def get_tools(self) -> List[Tool]: + """ + Get all tools from the repository. + + Returns: + A list of tools. + """ + pass + + @abstractmethod + async def get_tools_by_manual(self, manual_name: str) -> Optional[List[Tool]]: + """ + Get tools associated with a specific manual. + + Args: + manual_name: The name of the manual. + + Returns: + A list of tools associated with the manual, or None if the manual is not found. + """ + pass + + @abstractmethod + async def get_manual(self, manual_name: str) -> Optional[UtcpManual]: + """ + Get a manual from the repository. + + Args: + manual_name: The name of the manual to retrieve. + + Returns: + The manual if found, otherwise None. + """ + pass + + @abstractmethod + async def get_manuals(self) -> List[UtcpManual]: + """ + Get all manuals from the repository. + + Returns: + A list of manuals. + """ + pass + + @abstractmethod + async def get_manual_call_template(self, manual_call_template_name: str) -> Optional[CallTemplate]: + """ + Get a manual call template from the repository. + + Args: + manual_call_template_name: The name of the manual call template to retrieve. + + Returns: + The manual call template if found, otherwise None. + """ + pass + + @abstractmethod + async def get_manual_call_templates(self) -> List[CallTemplate]: + """ + Get all manual call templates from the repository. + + Returns: + A list of manual call templates. + """ + pass + +class ConcurrentToolRepositoryConfigSerializer(Serializer[ConcurrentToolRepository]): + tool_repository_implementations: Dict[str, Serializer['ConcurrentToolRepository']] = {} + default_repository = "in_memory" + + def to_dict(self, obj: ConcurrentToolRepository) -> dict: + return ConcurrentToolRepositoryConfigSerializer.tool_repository_implementations[obj.tool_repository_type].to_dict(obj) + + def validate_dict(self, data: dict) -> ConcurrentToolRepository: + try: + return ConcurrentToolRepositoryConfigSerializer.tool_repository_implementations[data['tool_repository_type']].validate_dict(data) + except KeyError: + raise ValueError(f"Invalid tool repository type: {data['tool_repository_type']}") + except Exception as e: + raise UtcpSerializerValidationError("Invalid ConcurrentToolRepository: " + traceback.format_exc()) from e diff --git a/core/src/utcp/interfaces/serializer.py b/core/src/utcp/interfaces/serializer.py new file mode 100644 index 0000000..a634a7b --- /dev/null +++ b/core/src/utcp/interfaces/serializer.py @@ -0,0 +1,21 @@ +from abc import ABC, abstractmethod +from typing import TypeVar, Generic +from utcp.plugins.plugin_loader import ensure_plugins_initialized + +T = TypeVar('T') + +class Serializer(ABC, Generic[T]): + + def __init__(self): + ensure_plugins_initialized() + + @abstractmethod + def validate_dict(self, obj: dict) -> T: + pass + + @abstractmethod + def to_dict(self, obj: T) -> dict: + pass + + def copy(self, obj: T) -> T: + return self.validate_dict(self.to_dict(obj)) diff --git a/core/src/utcp/interfaces/tool_post_processor.py b/core/src/utcp/interfaces/tool_post_processor.py new file mode 100644 index 0000000..257c175 --- /dev/null +++ b/core/src/utcp/interfaces/tool_post_processor.py @@ -0,0 +1,30 @@ +from abc import ABC, abstractmethod +from utcp.utcp_client import UtcpClient +from utcp.data.tool import Tool +from utcp.data.call_template import CallTemplate +from typing import Any, Dict +from utcp.interfaces.serializer import Serializer +from pydantic import BaseModel +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +class ToolPostProcessor(BaseModel, ABC): + tool_post_processor_type: str + + @abstractmethod + def post_process(self, caller: 'UtcpClient', tool: Tool, manual_call_template: 'CallTemplate', result: Any) -> Any: + raise NotImplementedError + +class ToolPostProcessorConfigSerializer(Serializer[ToolPostProcessor]): + tool_post_processor_implementations: Dict[str, Serializer[ToolPostProcessor]] = {} + + def to_dict(self, obj: ToolPostProcessor) -> dict: + return ToolPostProcessorConfigSerializer.tool_post_processor_implementations[obj.tool_post_processor_type].to_dict(obj) + + def validate_dict(self, data: dict) -> ToolPostProcessor: + try: + return ToolPostProcessorConfigSerializer.tool_post_processor_implementations[data['tool_post_processor_type']].validate_dict(data) + except KeyError: + raise ValueError(f"Invalid tool post processor type: {data['tool_post_processor_type']}") + except Exception as e: + raise UtcpSerializerValidationError("Invalid ToolPostProcessor: " + traceback.format_exc()) from e diff --git a/core/src/utcp/interfaces/tool_search_strategy.py b/core/src/utcp/interfaces/tool_search_strategy.py new file mode 100644 index 0000000..bc6e93d --- /dev/null +++ b/core/src/utcp/interfaces/tool_search_strategy.py @@ -0,0 +1,72 @@ +"""Abstract interface for tool search strategies. + +This module defines the contract for implementing tool search and ranking +algorithms. Different strategies can implement various approaches such as +tag-based search, semantic search, or hybrid approaches. +""" + +from abc import ABC, abstractmethod +from typing import List, Optional, Dict +from utcp.data.tool import Tool +from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepository +from utcp.interfaces.serializer import Serializer +from pydantic import BaseModel +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +class ToolSearchStrategy(BaseModel, ABC): + """Abstract interface for tool search implementations. + + Defines the contract for tool search strategies that can be plugged into + the UTCP client. Different implementations can provide various search + algorithms such as tag-based matching, semantic similarity, or keyword + search. + + Search strategies are responsible for: + - Interpreting search queries + - Ranking tools by relevance + - Limiting results appropriately + - Providing consistent search behavior + """ + tool_search_strategy_type: str + + @abstractmethod + async def search_tools(self, tool_repository: ConcurrentToolRepository, query: str, limit: int = 10, any_of_tags_required: Optional[List[str]] = None) -> List[Tool]: + """Search for tools relevant to the query. + + Executes a search against the available tools and returns the most + relevant matches ranked by the strategy's scoring algorithm. + + Args: + tool_repository: The tool repository to search within. + query: The search query string. Format depends on the strategy + (e.g., keywords, tags, natural language). + limit: Maximum number of tools to return. Use 0 for no limit. + Strategies should respect this limit for performance. + any_of_tags_required: Optional list of tags where one of them must be present in the tool's tags + for it to be considered a match. + + Returns: + List of Tool objects ranked by relevance, limited to the + specified count. Empty list if no matches found. + + Raises: + ValueError: If the query format is invalid for this strategy. + RuntimeError: If the search operation fails unexpectedly. + """ + pass + +class ToolSearchStrategyConfigSerializer(Serializer[ToolSearchStrategy]): + tool_search_strategy_implementations: Dict[str, Serializer['ToolSearchStrategy']] = {} + default_strategy = "tag_and_description_word_match" + + def to_dict(self, obj: ToolSearchStrategy) -> dict: + return ToolSearchStrategyConfigSerializer.tool_search_strategy_implementations[obj.tool_search_strategy_type].to_dict(obj) + + def validate_dict(self, data: dict) -> ToolSearchStrategy: + try: + return ToolSearchStrategyConfigSerializer.tool_search_strategy_implementations[data['tool_search_strategy_type']].validate_dict(data) + except KeyError: + raise ValueError(f"Invalid tool search strategy type: {data['tool_search_strategy_type']}") + except Exception as e: + raise UtcpSerializerValidationError("Invalid ToolSearchStrategy: " + traceback.format_exc()) from e diff --git a/core/src/utcp/interfaces/variable_substitutor.py b/core/src/utcp/interfaces/variable_substitutor.py new file mode 100644 index 0000000..0641958 --- /dev/null +++ b/core/src/utcp/interfaces/variable_substitutor.py @@ -0,0 +1,44 @@ +from abc import ABC, abstractmethod +from typing import Any, Optional, List +from utcp.data.utcp_client_config import UtcpClientConfig + +class VariableSubstitutor(ABC): + """Abstract interface for variable substitution implementations. + + Defines the contract for variable substitution systems that can replace + placeholders in configuration data with actual values from various sources. + Implementations handle different variable resolution strategies and + source hierarchies. + """ + + @abstractmethod + def substitute(self, obj: dict | list | str, config: UtcpClientConfig, variable_namespace: Optional[str] = None) -> Any: + """Substitute variables in the given object. + + Args: + obj: Object containing potential variable references to substitute. + Can be dict, list, str, or any other type. + config: UTCP client configuration containing variable definitions + and loaders. + variable_namespace: Optional variable namespace. + + Returns: + Object with all variable references replaced by their values. + + Raises: + UtcpVariableNotFound: If a referenced variable cannot be resolved. + """ + pass + + @abstractmethod + def find_required_variables(self, obj: dict | list | str, variable_namespace: Optional[str] = None) -> List[str]: + """Find all variable references in the given object. + + Args: + obj: Object to scan for variable references. + variable_namespace: Optional variable namespace. + + Returns: + List of fully-qualified variable names found in the object. + """ + pass diff --git a/tests/__init__.py b/core/src/utcp/plugins/__init__.py similarity index 100% rename from tests/__init__.py rename to core/src/utcp/plugins/__init__.py diff --git a/core/src/utcp/plugins/discovery.py b/core/src/utcp/plugins/discovery.py new file mode 100644 index 0000000..7cc7918 --- /dev/null +++ b/core/src/utcp/plugins/discovery.py @@ -0,0 +1,60 @@ +from utcp.data.auth import Auth, AuthSerializer +from utcp.data.variable_loader import VariableLoader, VariableLoaderSerializer +from utcp.interfaces.serializer import Serializer +from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepository, ConcurrentToolRepositoryConfigSerializer +from utcp.interfaces.tool_search_strategy import ToolSearchStrategy, ToolSearchStrategyConfigSerializer +from utcp.interfaces.tool_post_processor import ToolPostProcessor, ToolPostProcessorConfigSerializer +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.data.call_template import CallTemplate, CallTemplateSerializer +import logging + +logger = logging.getLogger(__name__) + +def register_auth(auth_type: str, serializer: Serializer[Auth], override: bool = False) -> bool: + if not override and auth_type in AuthSerializer.auth_serializers: + return False + AuthSerializer.auth_serializers[auth_type] = serializer + logger.info("Registered auth type: " + auth_type) + return True + +def register_variable_loader(loader_type: str, serializer: Serializer[VariableLoader], override: bool = False) -> bool: + if not override and loader_type in VariableLoaderSerializer.loader_serializers: + return False + VariableLoaderSerializer.loader_serializers[loader_type] = serializer + logger.info("Registered variable loader type: " + loader_type) + return True + +def register_call_template(call_template_type: str, serializer: Serializer[CallTemplate], override: bool = False) -> bool: + if not override and call_template_type in CallTemplateSerializer.call_template_serializers: + return False + CallTemplateSerializer.call_template_serializers[call_template_type] = serializer + logger.info("Registered call template type: " + call_template_type) + return True + +def register_communication_protocol(communication_protocol_type: str, communication_protocol: CommunicationProtocol, override: bool = False) -> bool: + if not override and communication_protocol_type in CommunicationProtocol.communication_protocols: + return False + CommunicationProtocol.communication_protocols[communication_protocol_type] = communication_protocol + logger.info("Registered communication protocol type: " + communication_protocol_type) + return True + +def register_tool_repository(tool_repository_type: str, tool_repository: Serializer[ConcurrentToolRepository], override: bool = False) -> bool: + if not override and tool_repository_type in ConcurrentToolRepositoryConfigSerializer.tool_repository_implementations: + return False + ConcurrentToolRepositoryConfigSerializer.tool_repository_implementations[tool_repository_type] = tool_repository + logger.info("Registered tool repository type: " + tool_repository_type) + return True + +def register_tool_search_strategy(strategy_type: str, strategy: Serializer[ToolSearchStrategy], override: bool = False) -> bool: + if not override and strategy_type in ToolSearchStrategyConfigSerializer.tool_search_strategy_implementations: + return False + ToolSearchStrategyConfigSerializer.tool_search_strategy_implementations[strategy_type] = strategy + logger.info("Registered tool search strategy type: " + strategy_type) + return True + +def register_tool_post_processor(tool_post_processor_type: str, tool_post_processor: Serializer[ToolPostProcessor], override: bool = False) -> bool: + if not override and tool_post_processor_type in ToolPostProcessorConfigSerializer.tool_post_processor_implementations: + return False + ToolPostProcessorConfigSerializer.tool_post_processor_implementations[tool_post_processor_type] = tool_post_processor + logger.info("Registered tool post processor type: " + tool_post_processor_type) + return True diff --git a/core/src/utcp/plugins/plugin_loader.py b/core/src/utcp/plugins/plugin_loader.py new file mode 100644 index 0000000..6fa2cf6 --- /dev/null +++ b/core/src/utcp/plugins/plugin_loader.py @@ -0,0 +1,45 @@ +import importlib.metadata + +def _load_plugins(): + from utcp.plugins.discovery import register_auth, register_variable_loader, register_tool_repository, register_tool_search_strategy, register_tool_post_processor + from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepositoryConfigSerializer + from utcp.interfaces.tool_search_strategy import ToolSearchStrategyConfigSerializer + from utcp.implementations.in_mem_tool_repository import InMemToolRepositoryConfigSerializer + from utcp.implementations.tag_search import TagAndDescriptionWordMatchStrategyConfigSerializer + from utcp.data.auth_implementations import OAuth2AuthSerializer, BasicAuthSerializer, ApiKeyAuthSerializer + from utcp.data.variable_loader_implementations import DotEnvVariableLoaderSerializer + from utcp.implementations.post_processors import FilterDictPostProcessorConfigSerializer, LimitStringsPostProcessorConfigSerializer + + register_auth("oauth2", OAuth2AuthSerializer()) + register_auth("basic", BasicAuthSerializer()) + register_auth("api_key", ApiKeyAuthSerializer()) + + register_variable_loader("dotenv", DotEnvVariableLoaderSerializer()) + + register_tool_repository(ConcurrentToolRepositoryConfigSerializer.default_repository, InMemToolRepositoryConfigSerializer()) + + register_tool_search_strategy(ToolSearchStrategyConfigSerializer.default_strategy, TagAndDescriptionWordMatchStrategyConfigSerializer()) + + register_tool_post_processor("filter_dict", FilterDictPostProcessorConfigSerializer()) + register_tool_post_processor("limit_strings", LimitStringsPostProcessorConfigSerializer()) + + for ep in importlib.metadata.entry_points(group="utcp.plugins"): + register_func = ep.load() + register_func() + +plugins_initialized = False +loading_plugins = False + +def ensure_plugins_initialized(): + global plugins_initialized + global loading_plugins + if plugins_initialized: + return + if loading_plugins: + return + loading_plugins = True + try: + _load_plugins() + plugins_initialized = True + finally: + loading_plugins = False diff --git a/tests/client/__init__.py b/core/src/utcp/python_specific_tooling/__init__.py similarity index 100% rename from tests/client/__init__.py rename to core/src/utcp/python_specific_tooling/__init__.py diff --git a/core/src/utcp/python_specific_tooling/async_rwlock.py b/core/src/utcp/python_specific_tooling/async_rwlock.py new file mode 100644 index 0000000..8efc249 --- /dev/null +++ b/core/src/utcp/python_specific_tooling/async_rwlock.py @@ -0,0 +1,74 @@ +from __future__ import annotations + +import asyncio +from contextlib import asynccontextmanager + + +class AsyncRWLock: + """An asyncio-compatible reader-writer lock with writer preference. + + - Multiple readers can hold the lock concurrently. + - Writers acquire exclusive access. + - Writer preference via a turnstile prevents writer starvation. + """ + + def __init__(self) -> None: + self._readers = 0 + self._readers_lock = asyncio.Lock() # protects _readers counter + self._resource_lock = asyncio.Lock() # exclusive resource access + self._turnstile = asyncio.Lock() # blocks readers when a writer is waiting/active + self._writers_lock = asyncio.Lock() # serialize writers acquiring the turnstile + + async def acquire_read(self) -> None: + # Readers pass through the turnstile so queued writers can block new readers + await self._turnstile.acquire() + self._turnstile.release() + + await self._readers_lock.acquire() + try: + self._readers += 1 + if self._readers == 1: + # First reader locks the resource + await self._resource_lock.acquire() + finally: + self._readers_lock.release() + + async def release_read(self) -> None: + await self._readers_lock.acquire() + try: + self._readers -= 1 + if self._readers == 0: + # Last reader releases the resource + self._resource_lock.release() + finally: + self._readers_lock.release() + + async def acquire_write(self) -> None: + # Ensure only one writer at a time attempts to block readers + await self._writers_lock.acquire() + try: + await self._turnstile.acquire() + # Now block new readers and take the resource + await self._resource_lock.acquire() + finally: + self._writers_lock.release() + + async def release_write(self) -> None: + self._resource_lock.release() + self._turnstile.release() + + @asynccontextmanager + async def read(self): + await self.acquire_read() + try: + yield + finally: + await self.release_read() + + @asynccontextmanager + async def write(self): + await self.acquire_write() + try: + yield + finally: + await self.release_write() diff --git a/src/utcp/shared/tool.py b/core/src/utcp/python_specific_tooling/tool_decorator.py similarity index 77% rename from src/utcp/shared/tool.py rename to core/src/utcp/python_specific_tooling/tool_decorator.py index f7f80ce..a12ca0e 100644 --- a/src/utcp/shared/tool.py +++ b/core/src/utcp/python_specific_tooling/tool_decorator.py @@ -1,78 +1,8 @@ -"""Tool definitions and schema generation for UTCP. - -This module provides the core tool definition models and utilities for -automatic schema generation from Python functions. It supports both -manual tool definitions and decorator-based automatic tool creation. - -Key Components: - - Tool: The main tool definition model - - ToolInputOutputSchema: JSON Schema for tool inputs and outputs - - ToolContext: Global tool registry - - @utcp_tool: Decorator for automatic tool creation from functions - - Schema generation utilities for Python type hints -""" - import inspect from typing import Dict, Any, Optional, List, Set, Tuple, get_type_hints, get_origin, get_args, Union -from pydantic import BaseModel, Field -from utcp.shared.provider import ProviderUnion - - -class ToolInputOutputSchema(BaseModel): - """JSON Schema definition for tool inputs and outputs. - - Represents a JSON Schema object that defines the structure and validation - rules for tool parameters (inputs) or return values (outputs). Compatible - with JSON Schema Draft 7. - - Attributes: - type: The JSON Schema type (object, array, string, number, boolean, null). - properties: Dictionary of property definitions for object types. - required: List of required property names for object types. - description: Human-readable description of the schema. - title: Title for the schema. - items: Schema definition for array item types. - enum: List of allowed values for enumeration types. - minimum: Minimum value for numeric types. - maximum: Maximum value for numeric types. - format: String format specification (e.g., "date", "email"). None for strings. - """ - - type: str = Field(default="object") - properties: Dict[str, Any] = Field(default_factory=dict) - required: Optional[List[str]] = None - description: Optional[str] = None - title: Optional[str] = None - items: Optional[Dict[str, Any]] = None # For array types - enum: Optional[List[Any]] = None # For enum types - minimum: Optional[float] = None # For number types - maximum: Optional[float] = None # For number types - format: Optional[str] = None # For string formats - -class Tool(BaseModel): - """Definition of a UTCP tool. - - Represents a callable tool with its metadata, input/output schemas, - and provider configuration. Tools are the fundamental units of - functionality in the UTCP ecosystem. - - Attributes: - name: Unique identifier for the tool, typically in format "provider.tool_name". - description: Human-readable description of what the tool does. - inputs: JSON Schema defining the tool's input parameters. - outputs: JSON Schema defining the tool's return value structure. - tags: List of tags for categorization and search. - average_response_size: Optional hint about typical response size in bytes. - tool_provider: Provider configuration for accessing this tool. - """ - - name: str - description: str = "" - inputs: ToolInputOutputSchema = Field(default_factory=ToolInputOutputSchema) - outputs: ToolInputOutputSchema = Field(default_factory=ToolInputOutputSchema) - tags: List[str] = [] - average_response_size: Optional[int] = None - tool_provider: ProviderUnion +from pydantic import BaseModel +from utcp.data.tool import Tool, JsonSchema +from utcp.data.call_template import CallTemplate class ToolContext: """Global registry for UTCP tools. @@ -98,7 +28,7 @@ def add_tool(tool: Tool) -> None: Note: Prints registration information for debugging purposes. """ - print(f"Adding tool: {tool.name} with provider: {tool.tool_provider.name if tool.tool_provider else 'None'}") + print(f"Adding tool: {tool.name} with call template: {tool.tool_call_template.name if tool.tool_call_template else 'None'}") ToolContext.tools.append(tool) @staticmethod @@ -372,7 +302,7 @@ def type_to_json_schema(param_type, param_name: Optional[str] = None, param_desc return val -def generate_input_schema(func, title: Optional[str], description: Optional[str]) -> ToolInputOutputSchema: +def generate_input_schema(func, title: Optional[str], description: Optional[str]) -> JsonSchema: """Generate input schema for a function's parameters. Analyzes a function's signature and type hints to create a JSON Schema @@ -385,7 +315,7 @@ def generate_input_schema(func, title: Optional[str], description: Optional[str] description: Optional description for the schema. Returns: - ToolInputOutputSchema object describing the function's input parameters. + JSONSchema object describing the function's input parameters. Includes parameter types, required fields, and descriptions. """ sig = inspect.signature(func) @@ -409,7 +339,7 @@ def generate_input_schema(func, title: Optional[str], description: Optional[str] required.append(param_name) input_desc = "\n".join([f"{name}: {desc}" for name, desc in param_description.items() if desc]) - schema = ToolInputOutputSchema( + schema = JsonSchema( type="object", properties=properties, required=required, @@ -419,7 +349,7 @@ def generate_input_schema(func, title: Optional[str], description: Optional[str] return schema -def generate_output_schema(func, title: Optional[str], description: Optional[str]) -> ToolInputOutputSchema: +def generate_output_schema(func, title: Optional[str], description: Optional[str]) -> JsonSchema: """Generate output schema for a function's return value. Analyzes a function's return type annotation to create a JSON Schema @@ -432,7 +362,7 @@ def generate_output_schema(func, title: Optional[str], description: Optional[str description: Optional description for the schema. Returns: - ToolInputOutputSchema object describing the function's return value. + JSONSchema object describing the function's return value. Contains "result" property with the return type and description. """ type_hints = get_type_hints(func) @@ -454,7 +384,7 @@ def generate_output_schema(func, title: Optional[str], description: Optional[str "description": f"No return value for {func_name}" } - schema = ToolInputOutputSchema( + schema = JsonSchema( type="object", properties=properties, required=required, @@ -466,12 +396,12 @@ def generate_output_schema(func, title: Optional[str], description: Optional[str def utcp_tool( - tool_provider: ProviderUnion, + tool_call_template: CallTemplate, name: Optional[str] = None, description: Optional[str] = None, tags: Optional[List[str]] = ["utcp"], - inputs: Optional[ToolInputOutputSchema] = None, - outputs: Optional[ToolInputOutputSchema] = None, + inputs: Optional[JsonSchema] = None, + outputs: Optional[JsonSchema] = None, ): """Decorator to convert Python functions into UTCP tools. @@ -480,7 +410,7 @@ def utcp_tool( ToolContext for discovery. Args: - tool_provider: Provider configuration for accessing this tool. + tool_call_template: Call template for accessing this tool. name: Optional custom name for the tool. Defaults to function name. description: Optional description. Defaults to function docstring. tags: Optional list of tags for categorization. Defaults to ["utcp"]. @@ -490,30 +420,13 @@ def utcp_tool( Returns: Decorator function that transforms the target function into a UTCP tool. - Examples: - >>> @utcp_tool(HttpProvider(url="https://api.example.com")) - ... def get_weather(location: str) -> dict: - ... pass - - >>> @utcp_tool( - ... tool_provider=CliProvider(command_name="curl"), - ... name="fetch_url", - ... description="Fetch content from a URL", - ... tags=["http", "utility"] - ... ) - ... def fetch(url: str) -> str: - ... pass - Note: The decorated function gains additional attributes: - input(): Returns the input schema - - output(): Returns the output schema + - output(): Returns the output schema - tool_definition(): Returns the complete Tool object """ def decorator(func): - if tool_provider.name is None: - tool_provider.name = f"{func.__name__}_provider" - func_name = name or func.__name__ func_description = description or func.__doc__ or "" @@ -527,7 +440,7 @@ def get_tool_definition(): tags=tags, inputs=input_tool_schema, outputs=output_tool_schema, - tool_provider=tool_provider + tool_call_template=tool_call_template ) func.input = lambda: input_tool_schema diff --git a/src/utcp/version.py b/core/src/utcp/python_specific_tooling/version.py similarity index 60% rename from src/utcp/version.py rename to core/src/utcp/python_specific_tooling/version.py index a2f21ea..11fa081 100644 --- a/src/utcp/version.py +++ b/core/src/utcp/python_specific_tooling/version.py @@ -1,16 +1,21 @@ from importlib.metadata import version, PackageNotFoundError import tomli from pathlib import Path +import logging -__version__ = "0.2.3" +logger = logging.getLogger(__name__) + +__version__ = "1.0.0" try: __version__ = version("utcp") except PackageNotFoundError: try: - pyproject_path = Path(__file__).parent.parent.parent / "pyproject.toml" + pyproject_path = Path(__file__).parent.parent.parent.parent / "pyproject.toml" if pyproject_path.exists(): with open(pyproject_path, "rb") as f: pyproject_data = tomli.load(f) __version__ = pyproject_data.get("project", {}).get("version", __version__) + else: + logger.warning("pyproject.toml not found") except (ImportError, FileNotFoundError, KeyError): - pass + logger.warning("Failed to load version from pyproject.toml") diff --git a/core/src/utcp/utcp_client.py b/core/src/utcp/utcp_client.py new file mode 100644 index 0000000..69cd499 --- /dev/null +++ b/core/src/utcp/utcp_client.py @@ -0,0 +1,179 @@ +"""Main UTCP client implementation. + +This module provides the primary client interface for the Universal Tool Calling +Protocol. The UtcpClient class manages multiple transport implementations, +tool repositories, search strategies, and CallTemplate configurations. + +Key Features: + - Multi-transport support (HTTP, CLI, WebSocket, etc.) + - Dynamic CallTemplate registration and deregistration + - Tool discovery and search capabilities + - Variable substitution for configuration + - Pluggable tool repositories and search strategies +""" + +from abc import ABC, abstractmethod +from typing import Dict, Any, List, Union, Optional, AsyncGenerator, TYPE_CHECKING + +from utcp.data.call_template import CallTemplate +from utcp.data.tool import Tool +from utcp.data.register_manual_response import RegisterManualResult +from utcp.plugins.plugin_loader import ensure_plugins_initialized + +if TYPE_CHECKING: + from utcp.data.utcp_client_config import UtcpClientConfig + +class UtcpClient(ABC): + """Abstract interface for UTCP client implementations. + + Defines the core contract for UTCP clients, including CallTemplate management, + tool execution, search capabilities, and variable handling. This interface + allows for different client implementations while maintaining consistency. + + The interface supports: + - CallTemplate lifecycle management (register/deregister) + - Tool discovery and execution + - Tool search and filtering + - Configuration variable validation + """ + + def __init__( + self, + config: 'UtcpClientConfig', + root_dir: Optional[str] = None, + ): + self.config = config + self.root_dir = root_dir + + @classmethod + async def create( + cls, + root_dir: Optional[str] = None, + config: Optional[Union[str, Dict[str, Any], 'UtcpClientConfig']] = None, + ) -> 'UtcpClient': + """ + Create a new instance of UtcpClient. + + Args: + root_dir: The root directory for the client to resolve relative paths from. Defaults to the current working directory. + config: The configuration for the client. Can be a path to a configuration file, a dictionary, or UtcpClientConfig object. + tool_repository: The tool repository to use. Defaults to InMemToolRepository. + search_strategy: The tool search strategy to use. Defaults to TagSearchStrategy. + + Returns: + A new instance of UtcpClient. + """ + ensure_plugins_initialized() + from utcp.implementations.utcp_client_implementation import UtcpClientImplementation + return await UtcpClientImplementation.create( + root_dir=root_dir, + config=config + ) + + @abstractmethod + async def register_manual(self, manual_call_template: CallTemplate) -> RegisterManualResult: + """ + Register a tool CallTemplate and its tools. + + Args: + manual_call_template: The CallTemplate to register. + + Returns: + A RegisterManualResult object containing the registered CallTemplate and its tools. + """ + pass + + @abstractmethod + async def register_manuals(self, manual_call_templates: List[CallTemplate]) -> List[RegisterManualResult]: + """ + Register multiple tool CallTemplates and their tools. + + Args: + manual_call_templates: List of CallTemplates to register. + + Returns: + A list of RegisterManualResult objects containing the registered CallTemplates and their tools. Order is not preserved. + """ + pass + + @abstractmethod + async def deregister_manual(self, manual_call_template_name: str) -> bool: + """ + Deregister a tool CallTemplate. + + Args: + manual_call_template_name: The name of the CallTemplate to deregister. + + Returns: + True if the CallTemplate was deregistered, False otherwise. + """ + pass + + @abstractmethod + async def call_tool(self, tool_name: str, tool_args: Dict[str, Any]) -> Any: + """ + Call a tool. + + Args: + tool_name: The name of the tool to call. + tool_args: The arguments to pass to the tool. + + Returns: + The result of the tool call. + """ + pass + + @abstractmethod + async def call_tool_streaming(self, tool_name: str, tool_args: Dict[str, Any]) -> AsyncGenerator[Any, None]: + """ + Call a tool streamingly. + + Args: + tool_name: The name of the tool to call. + tool_args: The arguments to pass to the tool. + + Returns: + An async generator that yields the result of the tool call. + """ + pass + + @abstractmethod + async def search_tools(self, query: str, limit: int = 10, any_of_tags_required: Optional[List[str]] = None) -> List[Tool]: + """ + Search for tools relevant to the query. + + Args: + query: The search query. + limit: The maximum number of tools to return. 0 for no limit. + any_of_tags_required: Optional list of tags where one of them must be present in the tool's tags + + Returns: + A list of tools that match the search query. + """ + pass + + @abstractmethod + async def get_required_variables_for_manual_and_tools(self, manual_call_template: CallTemplate) -> List[str]: + """ + Get the required variables for a manual CallTemplate and its tools. + + Args: + manual_call_template: The manual CallTemplate. + + Returns: + A list of required variables for the manual CallTemplate and its tools. + """ + pass + + @abstractmethod + async def get_required_variables_for_registered_tool(self, tool_name: str) -> List[str]: + """ + Get the required variables for a registered tool. + + Args: + tool_name: The name of a registered tool. + + Returns: + A list of required variables for the tool. + """ + pass diff --git a/tests/client/transport_interfaces/__init__.py b/core/tests/__init__.py similarity index 100% rename from tests/client/transport_interfaces/__init__.py rename to core/tests/__init__.py diff --git a/core/tests/client/__init__.py b/core/tests/client/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/core/tests/client/test_utcp_client.py b/core/tests/client/test_utcp_client.py new file mode 100644 index 0000000..909c501 --- /dev/null +++ b/core/tests/client/test_utcp_client.py @@ -0,0 +1,719 @@ +import pytest +import pytest_asyncio +import asyncio +import json +import os +import tempfile +from typing import Dict, Any, List, Optional +from unittest.mock import MagicMock, AsyncMock, patch +from pydantic import Field +from utcp.data.utcp_manual import UtcpManual +from utcp.data.register_manual_response import RegisterManualResult +from utcp.implementations.utcp_client_implementation import UtcpClientImplementation +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.utcp_client import UtcpClient +from utcp.data.utcp_client_config import UtcpClientConfig +from utcp.exceptions import UtcpVariableNotFound, UtcpSerializerValidationError +from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepository +from utcp.implementations.in_mem_tool_repository import InMemToolRepository +from utcp.interfaces.tool_search_strategy import ToolSearchStrategy +from utcp.implementations.tag_search import TagAndDescriptionWordMatchStrategy +from utcp.interfaces.variable_substitutor import VariableSubstitutor +from utcp.implementations.default_variable_substitutor import DefaultVariableSubstitutor +from utcp.data.tool import Tool, JsonSchema +from utcp.data.call_template import CallTemplate +from utcp_http.http_call_template import HttpCallTemplate +from utcp_cli.cli_call_template import CliCallTemplate +from utcp.data.auth_implementations import ApiKeyAuth + + +class MockToolRepository(ConcurrentToolRepository): + """Mock tool repository for testing.""" + + tool_repository_type: str = "mock" + manuals: Dict[str, UtcpManual] = Field(default_factory=dict) + manual_call_templates: Dict[str, CallTemplate] = Field(default_factory=dict) + tools: Dict[str, Tool] = Field(default_factory=dict) + + async def save_manual(self, manual_call_template: CallTemplate, manual: UtcpManual) -> None: + self.manual_call_templates[manual_call_template.name] = manual_call_template + self.manuals[manual_call_template.name] = manual + for tool in manual.tools: + self.tools[tool.name] = tool + + async def remove_manual(self, manual_name: str) -> bool: + if manual_name not in self.manuals: + return False + manual = self.manuals[manual_name] + for tool in manual.tools: + if tool.name in self.tools: + del self.tools[tool.name] + del self.manuals[manual_name] + del self.manual_call_templates[manual_name] + return True + + async def get_tool(self, tool_name: str) -> Optional[Tool]: + return self.tools.get(tool_name) + + async def get_tools(self) -> List[Tool]: + return list(self.tools.values()) + + async def get_manual(self, manual_name: str) -> Optional[UtcpManual]: + return self.manuals.get(manual_name) + + async def get_manual_call_template(self, manual_name: str) -> Optional[CallTemplate]: + return self.manual_call_templates.get(manual_name) + + async def get_manual_call_templates(self) -> List[CallTemplate]: + return list(self.manual_call_templates.values()) + + async def remove_tool(self, tool_name: str) -> bool: + if tool_name in self.tools: + del self.tools[tool_name] + return True + return False + + async def get_tools_by_manual(self, manual_name: str) -> Optional[List[Tool]]: + if manual_name in self.manuals: + return self.manuals[manual_name].tools + return None + + async def get_manuals(self) -> List[UtcpManual]: + return list(self.manuals.values()) + + +class MockToolSearchStrategy(ToolSearchStrategy): + """Mock search strategy for testing.""" + + tool_repository: ConcurrentToolRepository + tool_search_strategy_type: str = "mock" + + async def search_tools(self, tool_repository: ConcurrentToolRepository, query: str, limit: int = 10, any_of_tags_required: Optional[List[str]] = None) -> List[Tool]: + tools = await self.tool_repository.get_tools() + # Simple mock search: return tools that contain the query in name or description + matched_tools = [ + tool for tool in tools + if query.lower() in tool.name.lower() or query.lower() in tool.description.lower() + ] + return matched_tools[:limit] if limit > 0 else matched_tools + + +class MockCommunicationProtocol(CommunicationProtocol): + """Mock transport for testing.""" + + def __init__(self, manual: UtcpManual = None, call_result: Any = "mock_result"): + self.manual = manual or UtcpManual(utcp_version="1.0", manual_version="1.0", tools=[]) + self.call_result = call_result + self.registered_manuals = [] + self.deregistered_manuals = [] + self.tool_calls = [] + + async def register_manual(self, caller: 'UtcpClient', manual_call_template: CallTemplate) -> RegisterManualResult: + self.registered_manuals.append(manual_call_template) + return RegisterManualResult(manual_call_template=manual_call_template, manual=self.manual, success=True, errors=[]) + + async def deregister_manual(self, caller: 'UtcpClient', manual_call_template: CallTemplate) -> None: + self.deregistered_manuals.append(manual_call_template) + + async def call_tool(self, caller: 'UtcpClient', tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + self.tool_calls.append((tool_name, tool_args, tool_call_template)) + return self.call_result + + async def call_tool_streaming(self, caller: 'UtcpClient', tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + yield self.call_result + + +@pytest_asyncio.fixture +async def mock_tool_repository(): + """Create a mock tool repository.""" + return MockToolRepository() + + +@pytest_asyncio.fixture +async def mock_search_strategy(mock_tool_repository): + """Create a mock search strategy.""" + return MockToolSearchStrategy(tool_repository=mock_tool_repository) + + +@pytest_asyncio.fixture +async def sample_tools(): + """Create sample tools for testing.""" + http_call_template = HttpCallTemplate( + name="test_http_provider", + url="https://api.example.com/tool", + http_method="POST", + call_template_type="http" + ) + + cli_call_template = CliCallTemplate( + name="test_cli_provider", + command_name="echo", + call_template_type="cli" + ) + + return [ + Tool( + name="http_tool", + description="HTTP test tool", + inputs=JsonSchema( + type="object", + properties={"param1": {"type": "string", "description": "Test parameter"}}, + required=["param1"] + ), + outputs=JsonSchema( + type="object", + properties={"result": {"type": "string", "description": "Test result"}} + ), + tags=["http", "test"], + tool_call_template=http_call_template + ), + Tool( + name="cli_tool", + description="CLI test tool", + inputs=JsonSchema( + type="object", + properties={"command": {"type": "string", "description": "Command to execute"}}, + required=["command"] + ), + outputs=JsonSchema( + type="object", + properties={"output": {"type": "string", "description": "Command output"}} + ), + tags=["cli", "test"], + tool_call_template=cli_call_template + ) + ] + + +@pytest_asyncio.fixture +async def utcp_client(): + """Fixture for UtcpClient.""" + return await UtcpClient.create() + + +class TestUtcpClient: + """Test the UtcpClient implementation.""" + + @pytest.mark.asyncio + async def test_init(self, utcp_client): + """Test UtcpClient initialization.""" + assert isinstance(utcp_client.config.tool_repository, InMemToolRepository) + assert isinstance(utcp_client.config.tool_search_strategy, TagAndDescriptionWordMatchStrategy) + assert isinstance(utcp_client.variable_substitutor, DefaultVariableSubstitutor) + + @pytest.mark.asyncio + async def test_create_with_defaults(self): + """Test creating UtcpClient with default parameters.""" + client = await UtcpClient.create() + + assert isinstance(client.config, UtcpClientConfig) + assert isinstance(client.config.tool_repository, InMemToolRepository) + assert isinstance(client.config.tool_search_strategy, TagAndDescriptionWordMatchStrategy) + assert isinstance(client.variable_substitutor, DefaultVariableSubstitutor) + + @pytest.mark.asyncio + async def test_create_with_dict_config(self): + """Test creating UtcpClient with dictionary config.""" + config_dict = { + "variables": {"TEST_VAR": "test_value"}, + "tool_repository": { + "tool_repository_type": "in_memory" + }, + "tool_search_strategy": { + "tool_search_strategy_type": "tag_and_description_word_match" + }, + "manual_call_templates": [], + "post_processing": [] + } + + client = await UtcpClient.create(config=config_dict) + assert client.config.variables == {"TEST_VAR": "test_value"} + + @pytest.mark.asyncio + async def test_create_with_utcp_config(self): + """Test creating UtcpClient with UtcpClientConfig object.""" + repo = InMemToolRepository() + config = UtcpClientConfig( + variables={"TEST_VAR": "test_value"}, + tool_repository=repo, + tool_search_strategy=TagAndDescriptionWordMatchStrategy(), + manual_call_templates=[], + post_processing=[] + ) + + client = await UtcpClient.create(config=config) + assert client.config is config + + @pytest.mark.asyncio + async def test_register_manual(self, utcp_client, sample_tools): + """Test registering a manual.""" + http_call_template = HttpCallTemplate( + name="test_manual", + url="https://api.example.com/tool", + http_method="POST", + call_template_type="http" + ) + + # Mock the communication protocol + manual = UtcpManual(utcp_version="1.0", manual_version="1.0", tools=sample_tools[:1]) + mock_protocol = MockCommunicationProtocol(manual) + CommunicationProtocol.communication_protocols["http"] = mock_protocol + + result = await utcp_client.register_manual(http_call_template) + + assert result.success + assert len(result.manual.tools) == 1 + assert result.manual.tools[0].name == "test_manual.http_tool" # Should be prefixed + + registered_manual_template = mock_protocol.registered_manuals[0] + assert registered_manual_template.name == "test_manual" + + # Verify tool was saved in repository + saved_tool = await utcp_client.config.tool_repository.get_tool("test_manual.http_tool") + assert saved_tool is not None + + @pytest.mark.asyncio + async def test_register_manual_unsupported_type(self, utcp_client): + """Test registering a manual with unsupported type.""" + + with pytest.raises(Exception): + call_template = HttpCallTemplate( + name="test_manual", + url="https://example.com", + http_method="GET", + call_template_type="unsupported_type" + ) + await utcp_client.register_manual(call_template) + + @pytest.mark.asyncio + async def test_register_manual_name_sanitization(self, utcp_client, sample_tools): + """Test that manual names are sanitized.""" + call_template = HttpCallTemplate( + name="test-manual.with/special@chars", + url="https://api.example.com/tool", + http_method="POST", + call_template_type="http" + ) + + manual = UtcpManual(utcp_version="1.0", manual_version="1.0", tools=sample_tools[:1]) + mock_protocol = MockCommunicationProtocol(manual) + CommunicationProtocol.communication_protocols["http"] = mock_protocol + + result = await utcp_client.register_manual(call_template) + + # Name should be sanitized + assert result.manual_call_template.name == "test_manual_with_special_chars" + assert result.manual.tools[0].name == "test_manual_with_special_chars.http_tool" + + @pytest.mark.asyncio + async def test_deregister_manual(self, utcp_client, sample_tools): + """Test deregistering a manual.""" + call_template = HttpCallTemplate( + name="test_manual", + url="https://api.example.com/tool", + http_method="POST", + call_template_type="http" + ) + + manual = UtcpManual(utcp_version="1.0", manual_version="1.0", tools=sample_tools[:1]) + mock_protocol = MockCommunicationProtocol(manual) + CommunicationProtocol.communication_protocols["http"] = mock_protocol + + # First register the manual + await utcp_client.register_manual(call_template) + + # Then deregister it + result = await utcp_client.deregister_manual("test_manual") + assert result is True + + # Verify manual was removed from repository + saved_manual = await utcp_client.config.tool_repository.get_manual("test_manual") + assert saved_manual is None + + # Verify protocol deregister was called + assert len(mock_protocol.deregistered_manuals) == 1 + + @pytest.mark.asyncio + async def test_deregister_nonexistent_manual(self, utcp_client): + """Test deregistering a non-existent manual.""" + client = utcp_client + result = await client.deregister_manual("nonexistent") + assert result is False + + @pytest.mark.asyncio + async def test_call_tool(self, utcp_client, sample_tools): + """Test calling a tool.""" + client = utcp_client + call_template = HttpCallTemplate( + name="test_manual", + url="https://api.example.com/tool", + http_method="POST", + call_template_type="http" + ) + + manual = UtcpManual(utcp_version="1.0", manual_version="1.0", tools=sample_tools[:1]) + mock_protocol = MockCommunicationProtocol(manual, "test_result") + CommunicationProtocol.communication_protocols["http"] = mock_protocol + + # Register the manual first + await client.register_manual(call_template) + + # Call the tool + result = await client.call_tool("test_manual.http_tool", {"param1": "value1"}) + + assert result == "test_result" + assert len(mock_protocol.tool_calls) == 1 + assert mock_protocol.tool_calls[0][0] == "test_manual.http_tool" + assert mock_protocol.tool_calls[0][1] == {"param1": "value1"} + + @pytest.mark.asyncio + async def test_call_tool_nonexistent_manual(self, utcp_client): + """Test calling a tool with nonexistent manual.""" + client = utcp_client + # This will fail at get_tool, not get_manual + with pytest.raises(ValueError, match="Tool not found: nonexistent.tool"): + await client.call_tool("nonexistent.tool", {"param": "value"}) + + @pytest.mark.asyncio + async def test_call_tool_nonexistent_tool(self, utcp_client, sample_tools): + """Test calling a nonexistent tool.""" + client = utcp_client + call_template = HttpCallTemplate( + name="test_manual", + url="https://api.example.com/tool", + http_method="POST", + call_template_type="http" + ) + + manual = UtcpManual(utcp_version="1.0", manual_version="1.0", tools=sample_tools[:1]) + mock_protocol = MockCommunicationProtocol(manual) + CommunicationProtocol.communication_protocols["http"] = mock_protocol + + # Register the manual first + await client.register_manual(call_template) + + with pytest.raises(ValueError, match="Tool not found: test_manual.nonexistent"): + await client.call_tool("test_manual.nonexistent", {"param": "value"}) + + @pytest.mark.asyncio + async def test_search_tools(self, utcp_client, sample_tools): + """Test searching for tools.""" + client = utcp_client + # Clear any existing manuals from other tests to ensure a clean slate + manual_names = [manual_call_template.name for manual_call_template in await client.config.tool_repository.get_manual_call_templates()] + for name in manual_names: + await client.deregister_manual(name) + + # Mock the communication protocols + mock_http_protocol = MockCommunicationProtocol(UtcpManual(utcp_version="1.0", manual_version="1.0", tools=[sample_tools[0]])) + mock_cli_protocol = MockCommunicationProtocol(UtcpManual(utcp_version="1.0", manual_version="1.0", tools=[sample_tools[1]])) + CommunicationProtocol.communication_protocols["http"] = mock_http_protocol + CommunicationProtocol.communication_protocols["cli"] = mock_cli_protocol + + # Register manuals to add tools to the repository + await client.register_manual(sample_tools[0].tool_call_template) + await client.register_manual(sample_tools[1].tool_call_template) + + # Search for tools + results = await client.search_tools("http", limit=10) + + # Should find the HTTP tool + assert len(results) == 2 + assert "http" in results[0].name.lower() or "http" in results[0].description.lower() + + @pytest.mark.asyncio + async def test_get_required_variables_for_manual_and_tools(self, utcp_client): + """Test getting required variables for a manual.""" + client = utcp_client + call_template = HttpCallTemplate( + name="test_manual", + url="https://api.example.com/$API_URL", + http_method="POST", + auth=ApiKeyAuth(api_key="$API_KEY", var_name="Authorization"), + call_template_type="http" + ) + + # Mock the communication protocol to return an empty manual + mock_protocol = MockCommunicationProtocol(UtcpManual(utcp_version="1.0", manual_version="1.0", tools=[])) + CommunicationProtocol.communication_protocols["http"] = mock_protocol + + variables = await client.get_required_variables_for_manual_and_tools(call_template) + + # Using set because order doesn't matter + assert set(variables) == {"test__manual_API_URL", "test__manual_API_KEY"} + + @pytest.mark.asyncio + async def test_get_required_variables_for_registered_tool(self, utcp_client, sample_tools): + """Test getting required variables for a registered tool.""" + client = utcp_client + call_template = HttpCallTemplate( + name="test_manual", + url="https://api.example.com/$API_URL", + http_method="POST", + call_template_type="http" + ) + + tool = sample_tools[0] + tool.name = "test_manual.http_tool" + tool.tool_call_template = call_template + + # Add tool to repository + manual = UtcpManual(utcp_version="1.0", manual_version="1.0", tools=[tool]) + await client.config.tool_repository.save_manual(call_template, manual) + + variables = await client.get_required_variables_for_registered_tool("test_manual.http_tool") + + assert variables == ["test__manual_API_URL"] + + @pytest.mark.asyncio + async def test_get_required_variables_for_nonexistent_tool(self, utcp_client): + """Test getting required variables for a nonexistent tool.""" + client = utcp_client + with pytest.raises(ValueError, match="Tool not found: nonexistent.tool"): + await client.get_required_variables_for_registered_tool("nonexistent.tool") + + +class TestUtcpClientManualCallTemplateLoading: + """Test call template loading functionality.""" + + @pytest.mark.asyncio + async def test_load_manual_call_templates_from_file(self): + """Test loading call templates from a JSON file.""" + config_data = { + "manual_call_templates": [ + { + "name": "http_template", + "call_template_type": "http", + "url": "https://api.example.com/tools", + "http_method": "GET" + }, + { + "name": "cli_template", + "call_template_type": "cli", + "command_name": "echo" + } + ] + } + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(config_data, f) + temp_file = f.name + + try: + # Mock the communication protocols + mock_http_protocol = MockCommunicationProtocol() + mock_cli_protocol = MockCommunicationProtocol() + CommunicationProtocol.communication_protocols["http"] = mock_http_protocol + CommunicationProtocol.communication_protocols["cli"] = mock_cli_protocol + + # Re-create client with the config file to load templates + client = await UtcpClient.create(config=temp_file) + + assert len(client.config.manual_call_templates) == 2 + assert len(mock_http_protocol.registered_manuals) == 1 + assert len(mock_cli_protocol.registered_manuals) == 1 + + finally: + os.unlink(temp_file) + + @pytest.mark.asyncio + async def test_load_manual_call_templates_file_not_found(self): + """Test loading call templates from a non-existent file.""" + with pytest.raises(ValueError, match="Invalid config file"): + await UtcpClient.create(config="nonexistent_file.json") + + @pytest.mark.asyncio + async def test_load_manual_call_templates_invalid_json(self): + """Test loading call templates from invalid JSON file.""" + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + f.write("{\"invalid_json\": }") + temp_file = f.name + + try: + with pytest.raises(ValueError, match="Invalid config file"): + await UtcpClient.create(config=temp_file) + finally: + os.unlink(temp_file) + + @pytest.mark.asyncio + async def test_load_manual_call_templates_with_variables(self): + """Test loading call templates with variable substitution.""" + config_data = { + "variables": { + "http__template_BASE_URL": "https://api.example.com", + "http__template_API_KEY": "secret_key" + }, + "manual_call_templates": [ + { + "name": "http_template", + "call_template_type": "http", + "url": "$BASE_URL/tools", + "http_method": "GET", + "auth": { + "auth_type": "api_key", + "api_key": "$API_KEY", + "var_name": "Authorization" + } + } + ], + "tool_repository": { + "tool_repository_type": "in_memory" + }, + "tool_search_strategy": { + "tool_search_strategy_type": "tag_and_description_word_match" + } + } + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(config_data, f) + temp_file = f.name + + try: + # Mock the communication protocol + mock_protocol = MockCommunicationProtocol() + CommunicationProtocol.communication_protocols["http"] = mock_protocol + + # Create client with config file + client = await UtcpClient.create(config=temp_file) + + # Check that the registered call template has substituted values + registered_template = mock_protocol.registered_manuals[0] + assert registered_template.url == "https://api.example.com/tools" + assert registered_template.auth.api_key == "secret_key" + + finally: + os.unlink(temp_file) + + @pytest.mark.asyncio + async def test_load_manual_call_templates_missing_variable(self): + """Test loading call templates with missing variable.""" + config_data = { + "manual_call_templates": [{ + "name": "http_template", + "call_template_type": "http", + "url": "$MISSING_VAR/tools", + "http_method": "GET" + }] + } + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(config_data, f) + temp_file = f.name + + try: + with pytest.raises(UtcpVariableNotFound, match="Variable http__template_MISSING_VAR referenced in provider configuration not found"): + await UtcpClient.create(config=temp_file) + finally: + os.unlink(temp_file) + + +class TestUtcpClientCommunicationProtocols: + """Test communication protocol-related functionality.""" + + @pytest.mark.asyncio + async def test_variable_substitution(self, utcp_client): + """Test variable substitution in call templates.""" + client = utcp_client + call_template = HttpCallTemplate( + name="test_template", + url="$BASE_URL/api", + http_method="POST", + auth=ApiKeyAuth(api_key="$API_KEY", var_name="Authorization") + ) + + # Set up variables with call template prefix + client.config.variables = { + "test__template_BASE_URL": "https://api.example.com", + "test__template_API_KEY": "secret_key" + } + + substituted_template = client._substitute_call_template_variables(call_template, "test_template") + + assert substituted_template.url == "https://api.example.com/api" + assert substituted_template.auth.api_key == "secret_key" + + @pytest.mark.asyncio + async def test_variable_substitution_missing_variable(self, utcp_client): + """Test variable substitution with missing variable.""" + client = utcp_client + call_template = HttpCallTemplate( + name="test_template", + url="$MISSING_VAR/api", + http_method="POST" + ) + + with pytest.raises(UtcpVariableNotFound, match="Variable test__template_MISSING_VAR referenced in provider configuration not found"): + client._substitute_call_template_variables(call_template, "test_template") + + +class TestUtcpClientEdgeCases: + """Test edge cases and error conditions.""" + + @pytest.mark.asyncio + async def test_empty_call_template_file(self): + """Test loading an empty call template file.""" + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump({"manual_call_templates": []}, f) # Empty array + temp_file = f.name + + try: + client = await UtcpClient.create(config=temp_file) + assert client.config.manual_call_templates == [] + finally: + os.unlink(temp_file) + + @pytest.mark.asyncio + async def test_register_manual_with_existing_name(self, utcp_client): + """Test registering a manual with an existing name should raise an error.""" + client = utcp_client + template1 = HttpCallTemplate( + name="duplicate_name", + url="https://api.example1.com/tool", + http_method="POST", + call_template_type="http" + ) + template2 = HttpCallTemplate( + name="duplicate_name", + url="https://api.example2.com/tool", + http_method="GET", + call_template_type="http" + ) + + mock_protocol = MockCommunicationProtocol() + CommunicationProtocol.communication_protocols["http"] = mock_protocol + + # Register first manual + await client.register_manual(template1) + + # Attempting to register second manual with same name should raise an error + with pytest.raises(ValueError, match="Manual duplicate_name already registered"): + await client.register_manual(template2) + + # Should still have the first manual + saved_template = await client.config.tool_repository.get_manual_call_template("duplicate_name") + assert saved_template.url == "https://api.example1.com/tool" + assert saved_template.http_method == "POST" + + @pytest.mark.asyncio + async def test_load_call_templates_wrong_format(self): + """Test loading call templates with wrong JSON format (object instead of array).""" + # This is not a valid config, `manual_call_templates` should be a list + config_data = { + "manual_call_templates": { + "http_template": { + "call_template_type": "http", + "url": "https://api.example.com/tools", + "http_method": "GET" + } + } + } + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(config_data, f) + temp_file = f.name + + try: + with pytest.raises(UtcpSerializerValidationError): + await UtcpClient.create(config=temp_file) + finally: + os.unlink(temp_file) diff --git a/example/README.md b/example/README.md deleted file mode 100644 index d8eaf62..0000000 --- a/example/README.md +++ /dev/null @@ -1 +0,0 @@ -For full examples please head to the [UTCP Examples Repo](https://github.com/universal-tool-calling-protocol/utcp-examples) \ No newline at end of file diff --git a/example/requirements.txt b/example/requirements.txt deleted file mode 100644 index 208108d..0000000 --- a/example/requirements.txt +++ /dev/null @@ -1,8 +0,0 @@ -fastapi[standard] -uvicorn -httpx -pydantic -authlib -python-dotenv -openai -boto3>=1.28.0 diff --git a/example/src/full_llm_example/bedrock_utcp_example.py b/example/src/full_llm_example/bedrock_utcp_example.py deleted file mode 100644 index 7ee3c86..0000000 --- a/example/src/full_llm_example/bedrock_utcp_example.py +++ /dev/null @@ -1,384 +0,0 @@ -""" -UTCP Amazon Bedrock Integration Example - -This example demonstrates how to: -1. Initialize a UTCP client with tool providers from a config file -2. For each user request, search for relevant tools -3. Instruct Amazon Bedrock to respond with a tool call -4. Parse the tool call and execute it using the UTCP client -5. Return the results to Amazon Bedrock for a final response -""" - -import asyncio -import os -import json -import argparse -from pathlib import Path -from typing import Dict, Any, List, Tuple -import uuid -import traceback - -import boto3 -from dotenv import load_dotenv - -from utcp.client.utcp_client import UtcpClient -from utcp.client.utcp_client_config import UtcpClientConfig, UtcpDotEnv -from utcp.shared.tool import Tool - -# Global debug flag -DEBUG = False - -# Amazon Bedrock model ID -# modelId = 'us.anthropic.claude-3-7-sonnet-20250219-v1:0' -modelId = 'anthropic.claude-3-sonnet-20240229-v1:0' - -async def initialize_utcp_client() -> UtcpClient: - """Initialize the UTCP client with configuration.""" - config = UtcpClientConfig( - providers_file_path=str(Path(__file__).parent / "providers.json"), - load_variables_from=[ - UtcpDotEnv(env_file_path=str(Path(__file__).parent / ".env")) - ] - ) - - client = await UtcpClient.create(config) - return client - - -def format_tools_for_bedrock(tools: List[Tool]) -> Tuple[List[Dict[str, Any]], Dict[str, str]]: - """ - Convert UTCP tools to Bedrock tool format. - - Args: - tools: List of UTCP tools - - Returns: - Tuple containing: - - List of tools formatted for Bedrock - - Mapping between modified tool names and original names - """ - bedrock_tools = [] - tool_name_mapping = {} - - for tool in tools: - schema = tool.model_dump() - - # Create the input schema JSON - input_schema_json = { - "type": "object", - "properties": {}, - "required": [] - } - - # Add parameters to the input schema - if "parameters" in schema and "properties" in schema["parameters"]: - input_schema_json["properties"] = schema["parameters"]["properties"] - if "required" in schema["parameters"]: - input_schema_json["required"] = schema["parameters"]["required"] - - # Replace periods in tool name with underscores - original_name = tool.name - bedrock_tool_name = original_name.replace(".", "_") - - # Truncate if longer than 64 characters (Bedrock's limit) - if len(bedrock_tool_name) > 64: - short_uuid = str(uuid.uuid4())[:8] - short_name = f"{bedrock_tool_name[:55]}_{short_uuid}" - if DEBUG: - print(f"Tool name '{bedrock_tool_name}' is too long, using '{short_name}' instead") - bedrock_tool_name = short_name - - # Store the mapping between the modified name and original name - tool_name_mapping[bedrock_tool_name] = original_name - - # Format the tool for Bedrock - tool_spec = { - "name": bedrock_tool_name, - "description": tool.description, - "inputSchema": { - "json": input_schema_json - } - } - - bedrock_tools.append({"toolSpec": tool_spec}) - - return bedrock_tools, tool_name_mapping - - -async def get_bedrock_response(messages: List[Dict[str, str]], tools=None, system_prompt=None) -> Dict[str, Any]: - """ - Get a response from Amazon Bedrock using the Converse API. - - Args: - messages: List of conversation messages - tools: Optional list of tools formatted for Bedrock - system_prompt: Optional system prompt - - Returns: - Response from Bedrock Converse API - """ - bedrock_runtime = boto3.client('bedrock-runtime') - - # Add tools configuration if provided - tool_config = None - if tools: - tool_config = {"tools": tools} - if DEBUG: - print(f"Tool config: {json.dumps(tool_config, indent=2)}") - - # Prepare system prompt if provided - system = None - if system_prompt: - system = [{"text": system_prompt}] - - try: - # Build the API call parameters - converse_params = { - "modelId": modelId, - "messages": messages - } - - # Add optional parameters if provided - if tool_config: - converse_params["toolConfig"] = tool_config - - if system: - converse_params["system"] = system - - if DEBUG: - print("Calling Bedrock converse API...") - - response = bedrock_runtime.converse(**converse_params) - - if DEBUG: - print("Bedrock API call successful") - print(f"Response keys: {list(response.keys())}") - print(f"Response structure: {json.dumps(response, indent=2)}") - - return response - except Exception as e: - print(f"Error in get_bedrock_response: {str(e)}") - if DEBUG: - print(f"Traceback: {traceback.format_exc()}") - raise - - -def extract_text_from_content(content): - """ - Extract text from a content block or list of content blocks. - - Args: - content: Content block or list of content blocks - - Returns: - Extracted text or empty string if no text found - """ - if not content: - return "" - - if isinstance(content, list): - for item in content: - if isinstance(item, dict) and "text" in item: - return item["text"] - elif isinstance(item, str): - return item - return "" - elif isinstance(content, dict) and "text" in content: - return content["text"] - elif isinstance(content, str): - return content - - return "" - - -async def process_tool_calls(utcp_client, tool_use, tool_name_mapping): - """ - Process a tool call and execute it using the UTCP client. - - Args: - utcp_client: UTCP client instance - tool_use: Tool use information from Bedrock - tool_name_mapping: Mapping between modified tool names and original names - - Returns: - Dictionary containing tool result information - """ - tool_use_id = tool_use["toolUseId"] - modified_tool_name = tool_use["name"] - - # Map the modified tool name back to the original tool name - original_tool_name = tool_name_mapping.get(modified_tool_name, modified_tool_name) - - print(f"\nTool call detected: {original_tool_name}") - - # Get the tool arguments - tool_args = tool_use["input"] - print(f"Arguments: {json.dumps(tool_args, indent=2)}") - - try: - print(f"Executing tool call: {original_tool_name}") - result = await utcp_client.call_tool(original_tool_name, tool_args) - print(f"Tool execution successful!") - print(f"Result: {result}") - - # Format the tool result as expected by Bedrock - return { - "toolResult": { - "toolUseId": tool_use_id, - "content": [{"json": result}] - } - } - except Exception as e: - error_message = f"Error calling {original_tool_name}: {str(e)}" - print(f"Error: {error_message}") - - # Format the error as a tool result - return { - "toolResult": { - "toolUseId": tool_use_id, - "content": [{"json": {"error": str(e)}}] - } - } - - -async def main(): - """Main function to demonstrate Amazon Bedrock with UTCP integration.""" - load_dotenv(Path(__file__).parent / ".env") - - # Check for AWS credentials - if not (os.environ.get("AWS_ACCESS_KEY_ID") and os.environ.get("AWS_SECRET_ACCESS_KEY")): - print("Warning: AWS credentials not found in environment variables") - print("Make sure you have configured AWS credentials using AWS CLI or environment variables") - - print("Initializing UTCP client...") - utcp_client = await initialize_utcp_client() - print("UTCP client initialized successfully.") - print(f"Using model {modelId}") - - conversation_history = [] - tool_name_mapping = {} - - system_prompt = ( - "You are a helpful assistant with access to external tools. When a user asks a question that requires " - "using one of the available tools, you MUST use the appropriate tool rather than trying to answer from " - "your knowledge. Always prefer using tools when they are relevant to the query. " - "For example, if asked about news or books, use the corresponding tools to fetch real-time information. " - "When using a tool, analyze thoroughly the required tool parameters and pass them as required." - ) - - while True: - user_prompt = input("\nEnter your prompt (or 'exit' to quit): ") - if user_prompt.lower() in ["exit", "quit"]: - break - - print("\nSearching for relevant tools...") - relevant_tools = await utcp_client.search_tools(user_prompt, limit=10) - - if relevant_tools: - print(f"Found {len(relevant_tools)} relevant tools.") - for tool in relevant_tools: - print(f"- {tool.name}") - else: - print("No relevant tools found.") - - # Get the formatted tools and the mapping between modified and original names - bedrock_tools, name_mapping = format_tools_for_bedrock(relevant_tools) - tool_name_mapping.update(name_mapping) - - # Prepare messages for Bedrock - messages = conversation_history.copy() - messages.append({"role": "user", "content": [{"text": user_prompt}]}) - - print("\nSending request to Amazon Bedrock...") - try: - response = await get_bedrock_response(messages, bedrock_tools, system_prompt) - - # Process the response - if "output" not in response or "message" not in response["output"]: - print(f"Error: Unexpected response format. Missing 'output.message' key.") - if DEBUG: - print(f"Full response: {response}") - continue - - assistant_message = response["output"]["message"] - conversation_history.append({"role": "user", "content": [{"text": user_prompt}]}) - - # Check if the stop reason is tool_use - if response.get("stopReason") == "tool_use": - # Process tool use - tool_results = [] - - # Process each content block in the assistant's message - for content_block in assistant_message["content"]: - if "text" in content_block: - print(f"\nAssistant: {content_block['text']}") - - if "toolUse" in content_block: - tool_result = await process_tool_calls( - utcp_client, - content_block["toolUse"], - tool_name_mapping - ) - tool_results.append(tool_result) - - # Store assistant's response in conversation history - conversation_history.append(assistant_message) - - # Send the tool results back to Bedrock - print("\nSending tool results to Amazon Bedrock for interpretation...") - - # Prepare messages with tool results - tool_response_messages = messages.copy() - tool_response_messages.append(assistant_message) - tool_response_messages.append({ - "role": "user", - "content": tool_results - }) - - # Get final response from Bedrock - final_response = await get_bedrock_response(tool_response_messages, bedrock_tools, system_prompt) - - if "output" not in final_response or "message" not in final_response["output"]: - print(f"Error: Unexpected response format in final response.") - if DEBUG: - print(f"Full response: {final_response}") - continue - - final_message = final_response["output"]["message"] - final_text = extract_text_from_content(final_message.get("content", [])) - - print(f"\nAssistant's interpretation: {final_text}") - conversation_history.append({ - "role": "assistant", - "content": [{"text": final_text}] - }) - else: - # No tool call, just display the response - assistant_text = extract_text_from_content(assistant_message.get("content", [])) - if assistant_text: - print(f"\nAssistant: {assistant_text}") - conversation_history.append({ - "role": "assistant", - "content": [{"text": assistant_text}] - }) - else: - print(f"\nError: Unexpected assistant message format") - if DEBUG: - print(f"Message: {assistant_message}") - - except Exception as e: - print(f"Error calling Amazon Bedrock: {str(e)}") - if DEBUG: - print(f"Traceback: {traceback.format_exc()}") - - -if __name__ == "__main__": - # Parse command line arguments - parser = argparse.ArgumentParser(description="UTCP Amazon Bedrock Integration Example") - parser.add_argument("--debug", action="store_true", help="Enable debug output") - args = parser.parse_args() - - # Set global debug flag - DEBUG = args.debug - - asyncio.run(main()) diff --git a/example/src/full_llm_example/example.env b/example/src/full_llm_example/example.env deleted file mode 100644 index 4434899..0000000 --- a/example/src/full_llm_example/example.env +++ /dev/null @@ -1,12 +0,0 @@ -# News API credentials -NEWS_API_KEY=INSERT_YOUR_NEWS_API_KEY_HERE - -# OpenAI API credentials -OPENAI_API_KEY=INSERT_YOUR_OPENAI_API_KEY_HERE_AND_RENAME_THIS_FILE_TO_.env - -# AWS Credentials for Bedrock -AWS_ACCESS_KEY_ID=INSERT_YOUR_AWS_ACCESS_KEY_HERE -AWS_SECRET_ACCESS_KEY=INSERT_YOUR_AWS_SECRET_KEY_HERE -AWS_SESSION_TOKEN=INSERT_YOUR_AWS_SESSION_TOKEN_HERE_IF_USING_TEMPORARY_CREDENTIALS -AWS_REGION=us-east-1 -BEDROCK_MODEL_ID=anthropic.claude-3-sonnet-20240229-v1:0 \ No newline at end of file diff --git a/example/src/full_llm_example/newsapi_manual.json b/example/src/full_llm_example/newsapi_manual.json deleted file mode 100644 index 072fb1b..0000000 --- a/example/src/full_llm_example/newsapi_manual.json +++ /dev/null @@ -1,252 +0,0 @@ -{ - "version": "1.0", - "tools": [ - { - "name": "everything_get", - "description": "Search through millions of articles from over 150,000 large and small news sources and blogs. This endpoint suits article discovery and analysis. It requires either a search query, a source, or a domain.", - "tags": [ - "articles" - ], - "inputs": { - "type": "object", - "properties": { - "q": { - "type": "string", - "description": "Keywords or phrases to search for in the article title and body. Advanced search is supported. Max length: 500 chars." - }, - "searchIn": { - "type": "string", - "description": "The fields to restrict your q search to. Possible options: title, description, content. Multiple options can be specified by separating them with a comma." - }, - "sources": { - "type": "string", - "description": "A comma-seperated string of identifiers (maximum 20) for the news sources or blogs you want headlines from." - }, - "domains": { - "type": "string", - "description": "A comma-seperated string of domains (eg bbc.co.uk, techcrunch.com, engadget.com) to restrict the search to." - }, - "excludeDomains": { - "type": "string", - "description": "A comma-seperated string of domains (eg bbc.co.uk, techcrunch.com, engadget.com) to remove from the results." - }, - "from": { - "type": "string", - "description": "A date and optional time for the oldest article allowed. This should be in ISO 8601 format (e.g. 2025-07-09 or 2025-07-09T09:28:11)" - }, - "to": { - "type": "string", - "description": "A date and optional time for the newest article allowed. This should be in ISO 8601 format (e.g. 2025-07-09 or 2025-07-09T09:28:11)" - }, - "language": { - "type": "string", - "description": "The 2-letter ISO-639-1 code of the language you want to get headlines for." - }, - "sortBy": { - "type": "string", - "description": "The order to sort the articles in. Possible options: relevancy, popularity, publishedAt." - }, - "pageSize": { - "type": "integer", - "description": "The number of results to return per page. Maximum: 100." - }, - "page": { - "type": "integer", - "description": "Use this to page through the results." - } - }, - "required": [ - "q" - ] - }, - "outputs": { - "type": "object", - "properties": { - "status": { - "type": "string", - "description": "If the request was successful or not. Options: ok, error." - }, - "totalResults": { - "type": "integer", - "description": "The total number of results available for your request." - }, - "articles": { - "type": "array", - "description": "The results of the request.", - "items": { - "type": "object", - "properties": { - "source": { - "type": "object", - "description": "The identifier id and a display name name for the source this article came from.", - "properties": { - "id": { - "type": "string" - }, - "name": { - "type": "string" - } - } - }, - "author": { - "type": "string", - "description": "The author of the article" - }, - "title": { - "type": "string", - "description": "The headline or title of the article." - }, - "description": { - "type": "string", - "description": "A description or snippet from the article." - }, - "url": { - "type": "string", - "description": "The direct URL to the article." - }, - "urlToImage": { - "type": "string", - "description": "The URL to a relevant image for the article." - }, - "publishedAt": { - "type": "string", - "description": "The date and time that the article was published, in UTC (+000)" - }, - "content": { - "type": "string", - "description": "The unformatted content of the article, where available. This is truncated to 200 chars." - } - }, - "required": [ - "title" - ] - } - } - } - }, - "tool_provider": { - "provider_type": "http", - "url": "https://newsapi.org/v2/everything", - "http_method": "GET", - "content_type": "application/json", - "auth": { - "auth_type": "api_key", - "api_key": "$NEWS_API_KEY", - "var_name": "X-Api-Key" - } - } - }, - { - "name": "top_headlines_get", - "description": "This endpoint provides live top and breaking headlines for a country, specific category in a country, single source, or multiple sources. You can also search with keywords. Articles are sorted by the earliest date published first. This endpoint is great for retrieving headlines for use with news tickers or similar.", - "tags": [ - "articles" - ], - "inputs": { - "type": "object", - "properties": { - "country": { - "type": "string", - "description": "The 2-letter ISO 3166-1 code of the country you want to get headlines for. Note: you can't mix this param with the sources param." - }, - "category": { - "type": "string", - "description": "The category you want to get headlines for. Possible options: business, entertainment, general, health, science, sports, technology. Note: you can't mix this param with the sources param." - }, - "sources": { - "type": "string", - "description": "A comma-seperated string of identifiers for the news sources or blogs you want headlines from. Note: you can't mix this param with the country or category params." - }, - "q": { - "type": "string", - "description": "Keywords or a phrase to search for." - }, - "pageSize": { - "type": "integer", - "description": "The number of results to return per page (request). 20 is the default, 100 is the maximum." - }, - "page": { - "type": "integer", - "description": "Use this to page through the results if the total results found is greater than the page size." - } - } - }, - "outputs": { - "type": "object", - "properties": { - "status": { - "type": "string", - "description": "If the request was successful or not. Options: ok, error." - }, - "totalResults": { - "type": "integer", - "description": "The total number of results available for your request." - }, - "articles": { - "type": "array", - "description": "The results of the request.", - "items": { - "type": "object", - "properties": { - "source": { - "type": "object", - "description": "The identifier id and a display name name for the source this article came from.", - "properties": { - "id": { - "type": "string" - }, - "name": { - "type": "string" - } - } - }, - "author": { - "type": "string", - "description": "The author of the article" - }, - "title": { - "type": "string", - "description": "The headline or title of the article." - }, - "description": { - "type": "string", - "description": "A description or snippet from the article." - }, - "url": { - "type": "string", - "description": "The direct URL to the article." - }, - "urlToImage": { - "type": "string", - "description": "The URL to a relevant image for the article." - }, - "publishedAt": { - "type": "string", - "description": "The date and time that the article was published, in UTC (+000)" - }, - "content": { - "type": "string", - "description": "The unformatted content of the article, where available. This is truncated to 200 chars." - } - }, - "required": [ - "title" - ] - } - } - } - }, - "tool_provider": { - "provider_type": "http", - "url": "https://newsapi.org/v2/top-headlines", - "http_method": "GET", - "content_type": "application/json", - "auth": { - "auth_type": "api_key", - "api_key": "$NEWS_API_KEY", - "var_name": "X-Api-Key" - } - } - } - ] -} \ No newline at end of file diff --git a/example/src/full_llm_example/openai_utcp_example.py b/example/src/full_llm_example/openai_utcp_example.py deleted file mode 100644 index 0c7f1a2..0000000 --- a/example/src/full_llm_example/openai_utcp_example.py +++ /dev/null @@ -1,164 +0,0 @@ -""" -UTCP OpenAI Integration Example - -This example demonstrates how to: -1. Initialize a UTCP client with tool providers from a config file -2. For each user request, search for relevant tools. -3. Instruct OpenAI to respond with a JSON for a tool call. -4. Parse the JSON and execute the tool call using the UTCP client. -5. Return the results to OpenAI for a final response. -""" - -import asyncio -import os -import json -import sys -import re -from pathlib import Path -from typing import Dict, Any, List - -import openai -from dotenv import load_dotenv - -from utcp.client.utcp_client import UtcpClient -from utcp.client.utcp_client_config import UtcpClientConfig, UtcpDotEnv -from utcp.shared.tool import Tool - - -async def initialize_utcp_client() -> UtcpClient: - """Initialize the UTCP client with configuration.""" - # Create a configuration for the UTCP client - config = UtcpClientConfig( - providers_file_path=str(Path(__file__).parent / "providers.json"), - load_variables_from=[ - UtcpDotEnv(env_file_path=str(Path(__file__).parent / ".env")) - ] - ) - - # Create and return the UTCP client - client = await UtcpClient.create(config) - return client - -def format_tools_for_prompt(tools: List[Tool]) -> str: - """Convert UTCP tools to a JSON string for the prompt.""" - tool_list = [] - for tool in tools: - tool_list.append(tool.model_dump()) - return json.dumps(tool_list, indent=2) - -async def get_openai_response(messages: List[Dict[str, str]]) -> str: - """Get a response from OpenAI.""" - client = openai.AsyncOpenAI(api_key=os.environ.get("OPENAI_API_KEY")) - - response = await client.chat.completions.create( - model="gpt-4o-mini", - messages=messages, - ) - - return response.choices[0].message.content - -async def main(): - """Main function to demonstrate OpenAI with UTCP integration.""" - load_dotenv(Path(__file__).parent / ".env") - - if not os.environ.get("OPENAI_API_KEY"): - print("Error: OPENAI_API_KEY not found in environment variables") - print("Please set it in the .env file") - sys.exit(1) - - print("Initializing UTCP client...") - utcp_client = await initialize_utcp_client() - print("UTCP client initialized successfully.") - - conversation_history = [] - - while True: - user_prompt = input("\nEnter your prompt (or 'exit' to quit): ") - if user_prompt.lower() in ["exit", "quit"]: - break - - print("\nSearching for relevant tools...") - relevant_tools = await utcp_client.search_tools(user_prompt, limit=10) - - if relevant_tools: - print(f"Found {len(relevant_tools)} relevant tools.") - for tool in relevant_tools: - print(f"- {tool.name}") - else: - print("No relevant tools found.") - - tools_json_string = format_tools_for_prompt(relevant_tools) - - system_prompt = ( - "You are a helpful assistant. When you need to use a tool, you MUST respond with a JSON object " - "with 'tool_name' and 'arguments' keys. Do not add any other text. The arguments must be a JSON object." - "For example: {\"tool_name\": \"some_tool.name\", \"arguments\": {\"arg1\": \"value1\"}}. " - f"Here are the available tools:\n{tools_json_string}" - ) - - messages = [ - {"role": "system", "content": system_prompt}, - ] - if conversation_history: - messages.extend(conversation_history) - messages.append({"role": "user", "content": user_prompt}) - - print("\nSending request to OpenAI...") - assistant_response = await get_openai_response(messages) - - json_match = re.search(r'```json\n({.*?})\n```', assistant_response, re.DOTALL) - if not json_match: - json_match = re.search(r'({.*})', assistant_response, re.DOTALL) - - if json_match: - json_string = json_match.group(1) - try: - tool_call_data = json.loads(json_string) - if "tool_name" in tool_call_data and "arguments" in tool_call_data: - tool_name = tool_call_data["tool_name"] - arguments = tool_call_data["arguments"] - - print(f"\nExecuting tool call: {tool_name}") - print(f"Arguments: {json.dumps(arguments, indent=2)}") - - try: - result = await utcp_client.call_tool(tool_name, arguments) - print(f"Result: {result}") - tool_output = str(result) - except Exception as e: - error_message = f"Error calling {tool_name}: {str(e)}" - print(f"Error: {error_message}") - tool_output = error_message - - # Add user prompt and assistant's response to history - conversation_history.append({"role": "user", "content": user_prompt}) - conversation_history.append({"role": "assistant", "content": assistant_response}) - - print("\nSending tool results to OpenAI for interpretation...") - - # Create a new list of messages for the follow-up, adding the tool output as a new user message - follow_up_messages = [ - {"role": "system", "content": system_prompt}, - *conversation_history, - # Provide the tool's output as a new user message for the model to process - {"role": "user", "content": f"Tool output: {tool_output}\n Please use the tool output to answer the users request."} - ] - - final_response = await get_openai_response(follow_up_messages) - print(f"\nAssistant's interpretation: {final_response}") - conversation_history.append({"role": "assistant", "content": final_response}) - else: - print(f"\nAssistant: {assistant_response}") - conversation_history.append({"role": "user", "content": user_prompt}) - conversation_history.append({"role": "assistant", "content": assistant_response}) - except json.JSONDecodeError: - print(f"\nAssistant: {assistant_response}") - conversation_history.append({"role": "user", "content": user_prompt}) - conversation_history.append({"role": "assistant", "content": assistant_response}) - else: - print(f"\nAssistant: {assistant_response}") - conversation_history.append({"role": "user", "content": user_prompt}) - conversation_history.append({"role": "assistant", "content": assistant_response}) - -if __name__ == "__main__": - asyncio.run(main()) diff --git a/example/src/full_llm_example/providers.json b/example/src/full_llm_example/providers.json deleted file mode 100644 index d028922..0000000 --- a/example/src/full_llm_example/providers.json +++ /dev/null @@ -1,21 +0,0 @@ -[ - { - "name": "openlibrary", - "provider_type": "http", - "http_method": "GET", - "url": "https://openlibrary.org/static/openapi.json", - "content_type": "application/json" - }, - { - "name": "newsapi", - "provider_type": "text", - "file_path": "./newsapi_manual.json" - }, - { - "name": "openai", - "provider_type": "http", - "http_method": "GET", - "url": "https://raw.githubusercontent.com/openai/openai-openapi/refs/heads/manual_spec/openapi.yaml", - "content_type": "application/x-yaml" - } -] \ No newline at end of file diff --git a/example/src/simple_example/client.py b/example/src/simple_example/client.py deleted file mode 100644 index 58c38dc..0000000 --- a/example/src/simple_example/client.py +++ /dev/null @@ -1,25 +0,0 @@ -import asyncio -from os import getcwd -from utcp.client.utcp_client import UtcpClient - - -async def main(): - client: UtcpClient = await UtcpClient.create( - config={"providers_file_path": "./providers.json"} - ) - - # List all available tools - print("Registered tools:") - for tool in await client.tool_repository.get_tools(): - print(f" - {tool.name}") - - # Call one of the tools - tool_to_call = (await client.tool_repository.get_tools())[0].name - args = {"body": {"value": "test"}} - - result = await client.call_tool(tool_to_call, args) - print(f"\nTool call result for '{tool_to_call}':") - print(result) - -if __name__ == "__main__": - asyncio.run(main()) diff --git a/example/src/simple_example/providers.json b/example/src/simple_example/providers.json deleted file mode 100644 index a5db2ba..0000000 --- a/example/src/simple_example/providers.json +++ /dev/null @@ -1,9 +0,0 @@ -[ - { - "name": "test_provider", - "provider_type": "http", - "http_method": "GET", - "type": "utcp", - "url": "http://localhost:8080/utcp" - } -] \ No newline at end of file diff --git a/example/src/simple_example/server.py b/example/src/simple_example/server.py deleted file mode 100644 index 0bcb682..0000000 --- a/example/src/simple_example/server.py +++ /dev/null @@ -1,41 +0,0 @@ -from typing import List, Optional -from fastapi import FastAPI -from pydantic import BaseModel -from utcp.shared.provider import HttpProvider -from utcp.shared.tool import utcp_tool -from utcp.shared.utcp_manual import UtcpManual - -class TestInput(BaseModel): - value: str - -class TestRequest(BaseModel): - value: str - arr: List[TestInput] - -class TestResponse(BaseModel): - received: str - -__version__ = "1.0.0" -BASE_PATH = "http://localhost:8080" - -app = FastAPI() - -@app.get("/utcp", response_model=UtcpManual) -def get_utcp(): - return UtcpManual.create(version=__version__) - -@utcp_tool(tool_provider=HttpProvider( - name="test_provider", - url=f"{BASE_PATH}/test", - http_method="POST" -)) -@app.post("/test") -def test_endpoint(data: TestRequest) -> Optional[TestResponse]: - """Test endpoint to receive a string value. - - Args: - data (TestRequest): The input data containing a string value. - Returns: - TestResponse: A dictionary with the received value. - """ - return TestResponse(received=data.value) diff --git a/google_apis.json b/google_apis.json deleted file mode 100644 index ef09521..0000000 --- a/google_apis.json +++ /dev/null @@ -1,3093 +0,0 @@ -[ - { - "name": "googleapis.com:abusiveexperiencereport", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/abusiveexperiencereport/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:acceleratedmobilepageurl", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/acceleratedmobilepageurl/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:accessapproval", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/accessapproval/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:accesscontextmanager", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/accesscontextmanager/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:acmedns", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/acmedns/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:adexchangebuyer", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/adexchangebuyer/v1.4/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:adexchangebuyer2", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/adexchangebuyer2/v2beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:adexperiencereport", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/adexperiencereport/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:admin", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/admin/directory_v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:admob", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/admob/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:adsense", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/adsense/v1.4/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:adsensehost", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/adsensehost/v4.1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:advisorynotifications", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/advisorynotifications/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:alertcenter", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/alertcenter/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:analytics", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/analytics/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:analyticsadmin", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/analyticsadmin/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:analyticsdata", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/analyticsdata/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:analyticshub", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/analyticshub/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:analyticsreporting", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/analyticsreporting/v4/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:androiddeviceprovisioning", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/androiddeviceprovisioning/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:androidenterprise", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/androidenterprise/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:androidmanagement", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/androidmanagement/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:androidpublisher", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/androidpublisher/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:apigateway", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/apigateway/v1alpha2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:apigee", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/apigee/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:apigeeregistry", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/apigeeregistry/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:apikeys", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/apikeys/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:appengine", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/appengine/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:appsactivity", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/appsactivity/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:area120tables", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/area120tables/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:artifactregistry", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/artifactregistry/v1beta2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:assuredworkloads", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/assuredworkloads/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:authorizedbuyersmarketplace", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/authorizedbuyersmarketplace/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:automl", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/automl/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:baremetalsolution", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/baremetalsolution/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:batch", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/batch/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:beyondcorp", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/beyondcorp/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:bigquery", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/bigquery/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:bigqueryconnection", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/bigqueryconnection/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:bigquerydatatransfer", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/bigquerydatatransfer/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:bigqueryreservation", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/bigqueryreservation/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:bigtableadmin", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/bigtableadmin/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:billingbudgets", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/billingbudgets/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:binaryauthorization", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/binaryauthorization/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:blogger", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/blogger/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:books", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/books/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:businessprofileperformance", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/businessprofileperformance/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:calendar", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/calendar/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:certificatemanager", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/certificatemanager/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:chat", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/chat/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:chromemanagement", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/chromemanagement/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:chromepolicy", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/chromepolicy/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:chromeuxreport", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/chromeuxreport/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:civicinfo", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/civicinfo/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:classroom", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/classroom/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudasset", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudasset/v1p7beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudbilling", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudbilling/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudbuild", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudbuild/v1alpha2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudchannel", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudchannel/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:clouddebugger", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/clouddebugger/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:clouddeploy", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/clouddeploy/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:clouderrorreporting", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/clouderrorreporting/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudfunctions", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudfunctions/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudidentity", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudidentity/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudiot", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudiot/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudkms", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudkms/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudprivatecatalog", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudprivatecatalog/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudprivatecatalogproducer", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudprivatecatalogproducer/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudprofiler", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudprofiler/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudresourcemanager", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudresourcemanager/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudscheduler", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudscheduler/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudsearch", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudsearch/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudshell", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudshell/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudsupport", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudsupport/v2beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudtasks", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudtasks/v2beta3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:cloudtrace", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/cloudtrace/v2beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:commentanalyzer", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/commentanalyzer/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:composer", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/composer/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:compute", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/compute/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:connectors", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/connectors/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:contactcenteraiplatform", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/contactcenteraiplatform/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:contactcenterinsights", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/contactcenterinsights/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:container", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/container/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:containeranalysis", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/containeranalysis/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:content", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/content/v2.1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:contentwarehouse", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/contentwarehouse/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:customsearch", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/customsearch/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:datacatalog", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/datacatalog/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:dataflow", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/dataflow/v1b3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:dataform", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/dataform/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:datafusion", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/datafusion/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:datalabeling", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/datalabeling/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:datalineage", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/datalineage/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:datamigration", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/datamigration/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:datapipelines", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/datapipelines/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:dataplex", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/dataplex/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:dataproc", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/dataproc/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:datastore", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/datastore/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:datastream", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/datastream/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:deploymentmanager", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/deploymentmanager/v2beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:dfareporting", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/dfareporting/v3.4/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:dialogflow", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/dialogflow/v3beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:digitalassetlinks", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/digitalassetlinks/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:discovery", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/discovery/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:discoveryengine", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/discoveryengine/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:displayvideo", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/displayvideo/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:dlp", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/dlp/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:dns", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/dns/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:docs", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/docs/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:documentai", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/documentai/v1beta3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:domains", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/domains/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:domainsrdap", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/domainsrdap/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:doubleclickbidmanager", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/doubleclickbidmanager/v1.1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:doubleclicksearch", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/doubleclicksearch/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:drive", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/drive/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:driveactivity", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/driveactivity/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:drivelabels", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/drivelabels/v2beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:essentialcontacts", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/essentialcontacts/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:eventarc", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/eventarc/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:factchecktools", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/factchecktools/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:fcm", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/fcm/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:fcmdata", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/fcmdata/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:file", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/file/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebase", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebase/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebaseappcheck", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebaseappcheck/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebaseappdistribution", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebaseappdistribution/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebasedatabase", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebasedatabase/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebasedynamiclinks", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebasedynamiclinks/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebasehosting", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebasehosting/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebaseml", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebaseml/v1beta2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebaserules", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebaserules/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firebasestorage", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firebasestorage/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:firestore", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/firestore/v1beta2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:fitness", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/fitness/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:forms", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/forms/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:games", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/games/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:gamesConfiguration", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/gamesConfiguration/v1configuration/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:gamesManagement", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/gamesManagement/v1management/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:gameservices", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/gameservices/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:genomics", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/genomics/v2alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:gkebackup", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/gkebackup/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:gkehub", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/gkehub/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:gmail", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/gmail/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:gmailpostmastertools", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/gmailpostmastertools/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:groupsmigration", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/groupsmigration/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:groupssettings", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/groupssettings/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:healthcare", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/healthcare/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:homegraph", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/homegraph/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:iam", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/iam/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:iamcredentials", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/iamcredentials/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:iap", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/iap/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:ideahub", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/ideahub/v1alpha/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:identitytoolkit", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/identitytoolkit/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:ids", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/ids/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:indexing", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/indexing/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:integrations", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/integrations/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:jobs", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/jobs/v3p1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:keep", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/keep/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:kgsearch", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/kgsearch/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:kmsinventory", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/kmsinventory/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:language", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/language/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:libraryagent", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/libraryagent/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:licensing", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/licensing/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:lifesciences", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/lifesciences/v2beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:localservices", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/localservices/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:logging", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/logging/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:managedidentities", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/managedidentities/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:manufacturers", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/manufacturers/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:memcache", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/memcache/v1beta2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:metastore", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/metastore/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:migrationcenter", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/migrationcenter/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mirror", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mirror/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:ml", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/ml/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:monitoring", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/monitoring/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:my-business", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/my-business/v4/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mybusinessaccountmanagement", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mybusinessaccountmanagement/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mybusinessbusinesscalls", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mybusinessbusinesscalls/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mybusinessbusinessinformation", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mybusinessbusinessinformation/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mybusinesslodging", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mybusinesslodging/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mybusinessnotifications", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mybusinessnotifications/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mybusinessplaceactions", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mybusinessplaceactions/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mybusinessqanda", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mybusinessqanda/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:mybusinessverifications", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/mybusinessverifications/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:networkconnectivity", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/networkconnectivity/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:networkmanagement", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/networkmanagement/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:networksecurity", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/networksecurity/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:networkservices", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/networkservices/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:notebooks", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/notebooks/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:oauth2", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/oauth2/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:ondemandscanning", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/ondemandscanning/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:orgpolicy", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/orgpolicy/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:osconfig", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/osconfig/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:oslogin", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/oslogin/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:pagespeedonline", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/pagespeedonline/v5/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:paymentsresellersubscription", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/paymentsresellersubscription/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:people", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/people/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:playablelocations", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/playablelocations/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:playcustomapp", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/playcustomapp/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:playdeveloperreporting", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/playdeveloperreporting/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:playintegrity", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/playintegrity/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:plus", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/plus/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:policyanalyzer", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/policyanalyzer/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:policysimulator", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/policysimulator/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:policytroubleshooter", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/policytroubleshooter/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:poly", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/poly/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:privateca", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/privateca/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:prod_tt_sasportal", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/prod_tt_sasportal/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:proximitybeacon", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/proximitybeacon/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:publicca", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/publicca/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:pubsub", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/pubsub/v1beta2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:pubsublite", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/pubsublite/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:readerrevenuesubscriptionlinking", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/readerrevenuesubscriptionlinking/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:realtimebidding", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/realtimebidding/v1alpha/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:recaptchaenterprise", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/recaptchaenterprise/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:recommendationengine", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/recommendationengine/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:recommender", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/recommender/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:redis", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/redis/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:remotebuildexecution", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/remotebuildexecution/v1alpha/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:replicapool", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/replicapool/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:reseller", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/reseller/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:resourcesettings", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/resourcesettings/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:retail", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/retail/v2alpha/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:run", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/run/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:runtimeconfig", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/runtimeconfig/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:safebrowsing", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/safebrowsing/v4/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:sasportal", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/sasportal/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:script", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/script/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:searchads360", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/searchads360/v0/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:searchconsole", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/searchconsole/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:secretmanager", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/secretmanager/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:securitycenter", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/securitycenter/v1beta2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:servicebroker", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/servicebroker/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:serviceconsumermanagement", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/serviceconsumermanagement/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:servicecontrol", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/servicecontrol/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:servicedirectory", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/servicedirectory/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:servicemanagement", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/servicemanagement/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:servicenetworking", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/servicenetworking/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:serviceusage", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/serviceusage/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:sheets", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/sheets/v4/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:shoppingcontent", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/shoppingcontent/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:siteVerification", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/siteVerification/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:slides", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/slides/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:smartdevicemanagement", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/smartdevicemanagement/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:sourcerepo", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/sourcerepo/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:spanner", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/spanner/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:speech", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/speech/v2beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:sql", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/sql/v1beta4/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:sqladmin", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/sqladmin/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:storage", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/storage/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:storagetransfer", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/storagetransfer/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:streetviewpublish", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/streetviewpublish/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:sts", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/sts/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:tagmanager", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/tagmanager/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:tasks", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/tasks/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:testing", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/testing/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:texttospeech", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/texttospeech/v1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:toolresults", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/toolresults/v1beta3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:tpu", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/tpu/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:trafficdirector", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/trafficdirector/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:transcoder", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/transcoder/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:translate", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/translate/v3beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:travelimpactmodel", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/travelimpactmodel/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:vault", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/vault/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:vectortile", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/vectortile/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:verifiedaccess", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/verifiedaccess/v2/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:versionhistory", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/versionhistory/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:videointelligence", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/videointelligence/v1p3beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:vision", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/vision/v1p1beta1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:vmmigration", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/vmmigration/v1alpha1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:vpcaccess", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/vpcaccess/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:webfonts", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/webfonts/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:webmasters", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/webmasters/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:webrisk", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/webrisk/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:websecurityscanner", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/websecurityscanner/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:workflowexecutions", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/workflowexecutions/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:workflows", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/workflows/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:workloadmanager", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/workloadmanager/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:workstations", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/workstations/v1beta/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:youtube", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/youtube/v3/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:youtubeAnalytics", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/youtubeAnalytics/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - }, - { - "name": "googleapis.com:youtubereporting", - "provider_type": "http", - "http_method": "GET", - "url": "https://api.apis.guru/v2/specs/googleapis.com/youtubereporting/v1/openapi.json", - "content_type": "application/json", - "auth": null, - "headers": null, - "body_field": "body", - "header_fields": null - } -] \ No newline at end of file diff --git a/manual_creation_guide.md b/manual_creation_guide.md deleted file mode 100644 index 3b5563e..0000000 --- a/manual_creation_guide.md +++ /dev/null @@ -1,404 +0,0 @@ -# LLM Guide: Creating UTCP Manuals from API Specifications - -## 1. Objective - -Your task is to analyze a given API specification (e.g., OpenAPI/Swagger, or plain text documentation) and convert it into a `UTCPManual` JSON object. This manual allows a UTCP client to understand and interact with the API's tools. - -## 2. Core Concepts - -- **`UTCPManual`**: The root JSON object that contains a list of all available tools from a provider. It has two main keys: `version` and `tools`. -- **`Tool`**: A JSON object representing a single function or API endpoint. It describes what the tool does, what inputs it needs, what it returns, and how to call it. -- **`Provider`**: A JSON object *inside* a `Tool` that contains the specific connection details (e.g., HTTP URL, method, etc.). - -## 3. Step-by-Step Conversion Process - -Follow these steps to transform an API endpoint into a UTCP `Tool`. - -### Step 1: Identify Individual API Endpoints - -Scan the API documentation and treat each unique API endpoint as a separate tool. For a REST API, an endpoint is a unique combination of an HTTP method and a URL path (e.g., `GET /users/{id}` is one tool, and `POST /users` is another). - -### Step 2: For Each Endpoint, Create a `Tool` Object - -For every endpoint you identify, you will create one JSON object that will be added to the `tools` array in the final `UTCPManual`. - -### Step 3: Map API Details to `Tool` Fields - -This is the core of the task. Populate the fields of the `Tool` object as follows: - -- **`name`**: (String) Create a short, descriptive, `snake_case` name for the tool. Example: `get_user_by_id`. -- **`description`**: (String) Use the summary or description from the API documentation to explain what the tool does. -- **`tags`**: (Array of Strings) Add relevant keywords that can be used to search for this tool. These could be derived from the API's own tags or categories. Example: `["users", "profile", "read"]`. -- **`average_response_size`**: (Integer, Optional) If the API documentation provides information on the typical size of the response payload in bytes, include it here. This is useful for performance considerations. -- **`inputs`**: (Object) A JSON Schema object describing all the parameters the API endpoint accepts (path, query, headers, and body). - - Set `type` to `"object"`. - - In `properties`, create a key for *each* parameter. The value should be an object defining its `type` (e.g., `"string"`, `"number"`) and `description`. - - In `required`, create an array listing the names of all mandatory parameters. -- **`outputs`**: (Object) A JSON Schema object describing the successful response from the API (e.g., the `200 OK` response body). - - Set `type` to `"object"`. - - In `properties`, map the fields of the JSON response body. -- **`provider`**: (Object) This object contains the technical details needed to make the actual API call. - - `provider_type`: (String) Almost always `"http"` for web APIs. - - `url`: (String) The full URL of the endpoint. Use curly braces for path parameters, e.g., `https://api.example.com/users/{id}`. - - `http_method`: (String) The HTTP method, e.g., `"GET"`, `"POST"`. - - `content_type`: (String) The request's content type, typically `"application/json"`. - - `path_fields`: (Array of Strings) List the names of any parameters that are part of the URL path. - - `header_fields`: (Array of Strings) List the names of any parameters sent as request headers. - - `body_field`: (String) If the request has a JSON body, specify the name of the single input property that contains the body object. - - `auth`: (Object, Optional) If the API requires authentication, add this object. The `auth_type` field determines the authentication method (`api_key`, `basic`, or `oauth2`). Populate the other fields based on the API's security scheme. See the `auth.py` reference below for the exact structure. - -### Step 4: Assemble the Final `UTCPManual` - -Once you have created a `Tool` object for every endpoint, assemble them into the final `UTCPManual`. - -1. Create the root JSON object. -2. Set the `version` key to `"1.0"`. -3. Create a `tools` key with an array containing all the `Tool` objects you generated. - -## 4. Example - -## 5. Data Model Reference - -Below are the core Pydantic models that define the structure of a `UTCPManual`. Use these as the ground truth for the JSON structure you need to generate. - -### `tool.py` - -```python -import inspect -from typing import Dict, Any, Optional, List, Literal, Union, get_type_hints -from pydantic import BaseModel, Field, TypeAdapter -from utcp.shared.provider import ( - HttpProvider, - CliProvider, - WebSocketProvider, - GRPCProvider, - GraphQLProvider, - TCPProvider, - UDPProvider, - StreamableHttpProvider, - SSEProvider, - WebRTCProvider, - MCPProvider, - TextProvider, -) - -class ToolInputOutputSchema(BaseModel): - type: str = Field(default="object") - properties: Dict[str, Any] = Field(default_factory=dict) - required: Optional[List[str]] = None - description: Optional[str] = None - title: Optional[str] = None - -class Tool(BaseModel): - name: str - description: str = "" - inputs: ToolInputOutputSchema = Field(default_factory=ToolInputOutputSchema) - outputs: ToolInputOutputSchema = Field(default_factory=ToolInputOutputSchema) - tags: List[str] = [] - average_response_size: Optional[int] = None - provider: Optional[Union[ - HttpProvider, - CliProvider, - WebSocketProvider, - GRPCProvider, - GraphQLProvider, - TCPProvider, - UDPProvider, - StreamableHttpProvider, - SSEProvider, - WebRTCProvider, - MCPProvider, - TextProvider, - ]] = None -``` - -### `auth.py` - -```python -from typing import Literal, Optional, TypeAlias, Union - -from pydantic import BaseModel, Field - -class ApiKeyAuth(BaseModel): - """Authentication using an API key. - - The key can be provided directly or sourced from an environment variable. - """ - - auth_type: Literal["api_key"] = "api_key" - api_key: str = Field(..., description="The API key for authentication.") - var_name: str = Field( - ..., description="The name of the variable containing the API key." - ) - - -class BasicAuth(BaseModel): - """Authentication using a username and password.""" - - auth_type: Literal["basic"] = "basic" - username: str = Field(..., description="The username for basic authentication.") - password: str = Field(..., description="The password for basic authentication.") - - -class OAuth2Auth(BaseModel): - """Authentication using OAuth2.""" - - auth_type: Literal["oauth2"] = "oauth2" - token_url: str = Field(..., description="The URL to fetch the OAuth2 token from.") - client_id: str = Field(..., description="The OAuth2 client ID.") - client_secret: str = Field(..., description="The OAuth2 client secret.") - scope: Optional[str] = Field(None, description="The OAuth2 scope.") - - -Auth: TypeAlias = Union[ApiKeyAuth, BasicAuth, OAuth2Auth] -``` - -### `provider.py` - -```python -from typing import Dict, Any, Optional, List, Literal, TypeAlias, Union -from pydantic import BaseModel, Field - -from utcp.shared.auth import ( - Auth, - ApiKeyAuth, - BasicAuth, - OAuth2Auth, -) - -ProviderType: TypeAlias = Literal[ - 'http', # RESTful HTTP/HTTPS API - 'sse', # Server-Sent Events - 'http_stream', # HTTP Chunked Transfer Encoding - 'cli', # Command Line Interface - 'websocket', # WebSocket bidirectional connection - 'grpc', # gRPC (Google Remote Procedure Call) - 'graphql', # GraphQL query language - 'tcp', # Raw TCP socket - 'udp', # User Datagram Protocol - 'webrtc', # Web Real-Time Communication - 'mcp', # Model Context Protocol - 'text', # Text file provider -] - -class Provider(BaseModel): - name: str - provider_type: ProviderType - startup_command: Optional[List[str]] = None # For launching the provider if needed - -class HttpProvider(Provider): - """Options specific to HTTP tools""" - - provider_type: Literal["http"] = "http" - http_method: Literal["GET", "POST", "PUT", "DELETE", "PATCH"] = "GET" - url: str - content_type: str = "application/json" - auth: Optional[Auth] = None - headers: Optional[Dict[str, str]] = None - body_field: Optional[str] = Field(default=None, description="The name of the single input field to be sent as the request body.") - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers.") - -class SSEProvider(Provider): - """Options specific to Server-Sent Events tools""" - - provider_type: Literal["sse"] = "sse" - url: str - event_type: Optional[str] = None - reconnect: bool = True - retry_timeout: int = 30000 # Retry timeout in milliseconds if disconnected - auth: Optional[Auth] = None - headers: Optional[Dict[str, str]] = None - body_field: Optional[str] = Field(default=None, description="The name of the single input field to be sent as the request body.") - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers for the initial connection.") - -class StreamableHttpProvider(Provider): - """Options specific to HTTP Chunked Transfer Encoding (HTTP streaming) tools""" - - provider_type: Literal["http_stream"] = "http_stream" - url: str - http_method: Literal["GET", "POST"] = "GET" - content_type: str = "application/octet-stream" - chunk_size: int = 4096 # Size of chunks in bytes - timeout: int = 60000 # Timeout in milliseconds - headers: Optional[Dict[str, str]] = None - auth: Optional[Auth] = None - body_field: Optional[str] = Field(default=None, description="The name of the single input field to be sent as the request body.") - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers.") - -class CliProvider(Provider): - """Options specific to CLI tools""" - - provider_type: Literal["cli"] = "cli" - command_name: str - env_vars: Optional[Dict[str, str]] = Field(default=None, description="Environment variables to set when executing the command") - working_dir: Optional[str] = Field(default=None, description="Working directory for command execution") - auth: None = None - -class WebSocketProvider(Provider): - """Options specific to WebSocket tools""" - - provider_type: Literal["websocket"] = "websocket" - url: str - protocol: Optional[str] = None - keep_alive: bool = True - auth: Optional[Auth] = None - headers: Optional[Dict[str, str]] = None - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers for the initial connection.") - -class GRPCProvider(Provider): - """Options specific to gRPC tools""" - - provider_type: Literal["grpc"] = "grpc" - host: str - port: int - service_name: str - method_name: str - use_ssl: bool = False - auth: Optional[Auth] = None - -class GraphQLProvider(Provider): - """Options specific to GraphQL tools""" - - provider_type: Literal["graphql"] = "graphql" - url: str - operation_type: Literal["query", "mutation", "subscription"] = "query" - operation_name: Optional[str] = None - auth: Optional[Auth] = None - headers: Optional[Dict[str, str]] = None - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers for the initial connection.") - -class TCPProvider(Provider): - """Options specific to raw TCP socket tools""" - - provider_type: Literal["tcp"] = "tcp" - host: str - port: int - timeout: int = 30000 - auth: None = None - -class UDPProvider(Provider): - """Options specific to UDP socket tools""" - - provider_type: Literal["udp"] = "udp" - host: str - port: int - timeout: int = 30000 - auth: None = None - -class WebRTCProvider(Provider): - """Options specific to WebRTC tools""" - - provider_type: Literal["webrtc"] = "webrtc" - signaling_server: str - peer_id: str - data_channel_name: str = "tools" - auth: None = None - -class McpStdioServer(BaseModel): - """Configuration for an MCP server connected via stdio.""" - transport: Literal["stdio"] = "stdio" - command: str - args: Optional[List[str]] = [] - env: Optional[Dict[str, str]] = {} - -class McpHttpServer(BaseModel): - """Configuration for an MCP server connected via streamable HTTP.""" - transport: Literal["http"] = "http" - url: str - -McpServer: TypeAlias = Union[McpStdioServer, McpHttpServer] - -class McpConfig(BaseModel): - mcpServers: Dict[str, McpServer] - -class MCPProvider(Provider): - """Options specific to MCP tools, supporting both stdio and HTTP transports.""" - - provider_type: Literal["mcp"] = "mcp" - config: McpConfig - auth: Optional[OAuth2Auth] = None - - -class TextProvider(Provider): - """Options specific to text file-based tools. - - This provider reads tool definitions from a local text file. This is useful - when the tool call is included in the startup command, but the result of the - tool call produces a file at a static location that can be read from. It can - also be used as a UTCP tool provider to specify tools that should be used - from different other providers. - """ - - provider_type: Literal["text"] = "text" - file_path: str = Field(..., description="The path to the file containing the tool definitions.") - auth: None = None -``` - -**API Specification Snippet:** - -``` -Endpoint: GET /v1/weather -Description: Retrieves the current weather for a specific city. - -Query Parameters: -- `city` (string, required): The name of the city (e.g., "London"). -- `units` (string, optional): The temperature units. Can be 'metric' or 'imperial'. Defaults to 'metric'. - -Response (200 OK): -{ - "temperature": 15, - "conditions": "Cloudy", - "humidity": 82 -} -``` - -**Generated `UTCPManual`:** - -```json -{ - "version": "1.0", - "tools": [ - { - "name": "get_weather", - "description": "Retrieves the current weather for a specific city.", - "inputs": { - "type": "object", - "properties": { - "city": { - "type": "string", - "description": "The name of the city (e.g., \"London\")." - }, - "units": { - "type": "string", - "description": "The temperature units. Can be 'metric' or 'imperial'." - } - }, - "required": [ - "city" - ] - }, - "outputs": { - "type": "object", - "properties": { - "temperature": { - "type": "number" - }, - "conditions": { - "type": "string" - }, - "humidity": { - "type": "number" - } - } - }, - "tool_provider": { - "name": "weather_service", - "provider_type": "http", - "url": "https://api.example.com/v1/weather", - "http_method": "GET", - "content_type": "application/json" - } - } - ] -} -``` diff --git a/plugins/communication_protocols/cli/pyproject.toml b/plugins/communication_protocols/cli/pyproject.toml new file mode 100644 index 0000000..deb4ad6 --- /dev/null +++ b/plugins/communication_protocols/cli/pyproject.toml @@ -0,0 +1,43 @@ +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" + +[project] +name = "utcp-cli" +version = "1.0.0" +authors = [ + { name = "UTCP Contributors" }, +] +description = "Universal Tool Calling Protocol (UTCP) client library for Python" +readme = "README.md" +requires-python = ">=3.10" +dependencies = [ + "pydantic>=2.0", + "pyyaml>=6.0", + "utcp>=1.0" +] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "Programming Language :: Python :: 3", + "Operating System :: OS Independent", +] +license = "MPL-2.0" + +[project.optional-dependencies] +dev = [ + "build", + "pytest", + "pytest-asyncio", + "pytest-cov", + "coverage", + "twine", +] + +[project.urls] +Homepage = "https://utcp.io" +Source = "https://github.com/universal-tool-calling-protocol/python-utcp" +Issues = "https://github.com/universal-tool-calling-protocol/python-utcp/issues" + +[project.entry-points."utcp.plugins"] +cli = "utcp_cli:register" \ No newline at end of file diff --git a/plugins/communication_protocols/cli/src/utcp_cli/__init__.py b/plugins/communication_protocols/cli/src/utcp_cli/__init__.py new file mode 100644 index 0000000..a7bebd9 --- /dev/null +++ b/plugins/communication_protocols/cli/src/utcp_cli/__init__.py @@ -0,0 +1,13 @@ +from utcp.plugins.discovery import register_communication_protocol, register_call_template +from utcp_cli.cli_communication_protocol import CliCommunicationProtocol +from utcp_cli.cli_call_template import CliCallTemplate, CliCallTemplateSerializer + +def register(): + register_communication_protocol("cli", CliCommunicationProtocol()) + register_call_template("cli", CliCallTemplateSerializer()) + +__all__ = [ + "CliCommunicationProtocol", + "CliCallTemplate", + "CliCallTemplateSerializer", +] diff --git a/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py b/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py new file mode 100644 index 0000000..60ac21f --- /dev/null +++ b/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py @@ -0,0 +1,44 @@ +from typing import Optional, Dict, Literal +from pydantic import Field + +from utcp.data.call_template import CallTemplate +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +class CliCallTemplate(CallTemplate): + """Call template configuration for Command Line Interface tools. + + Enables execution of command-line tools and programs as UTCP providers. + Supports environment variable injection and custom working directories. + + Attributes: + call_template_type: Always "cli" for CLI providers. + command_name: The name or path of the command to execute. + env_vars: Optional environment variables to set during command execution. + working_dir: Optional custom working directory for command execution. + auth: Always None - CLI providers don't support authentication. + """ + + call_template_type: Literal["cli"] = "cli" + command_name: str + env_vars: Optional[Dict[str, str]] = Field( + default=None, description="Environment variables to set when executing the command" + ) + working_dir: Optional[str] = Field( + default=None, description="Working directory for command execution" + ) + auth: None = None + + +class CliCallTemplateSerializer(Serializer[CliCallTemplate]): + """Serializer for CliCallTemplate.""" + + def to_dict(self, obj: CliCallTemplate) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> CliCallTemplate: + try: + return CliCallTemplate.model_validate(obj) + except Exception as e: + raise UtcpSerializerValidationError("Invalid CliCallTemplate: " + traceback.format_exc()) from e diff --git a/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.py b/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.py new file mode 100644 index 0000000..48793e7 --- /dev/null +++ b/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.py @@ -0,0 +1,482 @@ +"""Command Line Interface (CLI) transport for UTCP client. + +This module provides the CLI transport implementation that enables UTCP clients +to interact with command-line tools and processes. It handles tool discovery +through startup commands, tool execution with proper argument formatting, +and output processing with JSON parsing capabilities. + +Key Features: + - Asynchronous command execution with timeout handling + - Tool discovery via startup commands that output UTCP manuals + - Flexible argument formatting for command-line flags + - Environment variable support for authentication and configuration + - JSON output parsing with fallback to raw text + - Cross-platform command parsing (Windows/Unix) + - Working directory control for command execution + +Security: + - Command execution is isolated through subprocess + - Environment variables can be controlled per provider + - Working directory can be restricted +""" +import asyncio +import json +import os +import shlex +from typing import Dict, Any, List, Optional, Callable, AsyncGenerator + +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.data.call_template import CallTemplate, CallTemplateSerializer +from utcp.data.tool import Tool +from utcp.data.utcp_manual import UtcpManual, UtcpManualSerializer +from utcp.data.register_manual_response import RegisterManualResult +from utcp_cli.cli_call_template import CliCallTemplate, CliCallTemplateSerializer +import logging + +logger = logging.getLogger(__name__) + + +class CliCommunicationProtocol(CommunicationProtocol): + """Transport implementation for CLI-based tool providers. + + Handles communication with command-line tools by executing processes + and managing their input/output. Supports both tool discovery and + execution phases with comprehensive error handling and timeout management. + + Features: + - Asynchronous subprocess execution with proper cleanup + - Tool discovery through startup commands returning UTCP manuals + - Flexible argument formatting for various CLI conventions + - Environment variable injection for authentication + - JSON output parsing with graceful fallback to text + - Cross-platform command parsing and execution + - Configurable working directories and timeouts + - Process lifecycle management with proper termination + + Architecture: + CLI tools are discovered by executing the provider's command_name + and parsing the output for UTCP manual JSON. Tool calls execute + the same command with formatted arguments and return processed output. + + Attributes: + _log: Logger function for debugging and error reporting. + """ + + def __init__(self): + """Initialize the CLI transport.""" + + def _log_info(self, message: str): + """Log informational messages.""" + logger.info(f"[CliCommunicationProtocol] {message}") + + def _log_error(self, message: str): + """Log error messages.""" + logger.error(f"[CliCommunicationProtocol Error] {message}") + + def _prepare_environment(self, provider: CliCallTemplate) -> Dict[str, str]: + """Prepare environment variables for command execution. + + Args: + provider: The CLI provider + + Returns: + Environment variables dictionary + """ + import os + env = os.environ.copy() + + # Add custom environment variables if provided + if provider.env_vars: + env.update(provider.env_vars) + + return env + + async def _execute_command( + self, + command: List[str], + env: Dict[str, str], + timeout: float = 30.0, + input_data: Optional[str] = None, + working_dir: Optional[str] = None + ) -> tuple[str, str, int]: + """Execute a command asynchronously. + + Args: + command: Command and arguments to execute + env: Environment variables + timeout: Timeout in seconds + input_data: Optional input data to pass to the command + working_dir: Working directory for command execution + + Returns: + Tuple of (stdout, stderr, return_code) + """ + process = None + try: + process = await asyncio.create_subprocess_exec( + *command, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + env=env, + cwd=working_dir, + stdin=asyncio.subprocess.PIPE if input_data else None + ) + + stdout_bytes, stderr_bytes = await asyncio.wait_for( + process.communicate(input=input_data.encode('utf-8') if input_data else None), + timeout=timeout + ) + + stdout = stdout_bytes.decode('utf-8', errors='replace') + stderr = stderr_bytes.decode('utf-8', errors='replace') + + return stdout, stderr, process.returncode or 0 + + except asyncio.TimeoutError: + # Kill the process if it times out + if process: + try: + process.kill() + await process.wait() + except ProcessLookupError: + pass # Process already terminated + self._log_error(f"Command timed out after {timeout} seconds: {' '.join(command)}") + raise + except Exception as e: + # Ensure process is cleaned up on any error + if process: + try: + process.kill() + await process.wait() + except ProcessLookupError: + pass # Process already terminated + self._log_error(f"Error executing command {' '.join(command)}: {e}") + raise + + async def register_manual(self, caller, manual_call_template: CallTemplate) -> RegisterManualResult: + """Register a CLI manual and discover its tools. + + Executes the call template's command_name and looks for a UTCP manual JSON in the output. + """ + if not isinstance(manual_call_template, CliCallTemplate): + raise ValueError("CliCommunicationProtocol can only be used with CliCallTemplate") + + if not manual_call_template.command_name: + raise ValueError(f"CliCallTemplate '{manual_call_template.name}' must have command_name set") + + self._log_info( + f"Registering CLI manual '{manual_call_template.name}' with command '{manual_call_template.command_name}'" + ) + + try: + env = self._prepare_environment(manual_call_template) + # Parse command string into proper arguments + # Use posix=False on Windows, posix=True on Unix-like systems + command = shlex.split(manual_call_template.command_name, posix=(os.name != 'nt')) + + self._log_info(f"Executing command for tool discovery: {' '.join(command)}") + + stdout, stderr, return_code = await self._execute_command( + command, + env, + timeout=30.0, + working_dir=manual_call_template.working_dir, + ) + + # Get output based on exit code + output = stdout if return_code == 0 else stderr + + if not output.strip(): + self._log_info( + f"No output from command '{manual_call_template.command_name}'" + ) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[ + f"No output from discovery command for CLI provider '{manual_call_template.name}'" + ], + ) + + # Try to parse UTCPManual from the output + utcp_manual = self._extract_utcp_manual_from_output( + output, manual_call_template.name + ) + + if utcp_manual is None: + error_msg = ( + f"Could not parse UTCP manual from CLI provider '{manual_call_template.name}' output" + ) + self._log_error(error_msg) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[error_msg], + ) + + self._log_info( + f"Discovered {len(utcp_manual.tools)} tools from CLI provider '{manual_call_template.name}'" + ) + return RegisterManualResult( + success=True, + manual_call_template=manual_call_template, + manual=utcp_manual, + errors=[], + ) + + except Exception as e: + error_msg = f"Error discovering tools from CLI provider '{manual_call_template.name}': {e}" + self._log_error(error_msg) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[error_msg], + ) + + async def deregister_manual(self, caller, manual_call_template: CallTemplate) -> None: + """Deregister a CLI manual (no-op).""" + if isinstance(manual_call_template, CliCallTemplate): + self._log_info( + f"Deregistering CLI manual '{manual_call_template.name}' (no-op)" + ) + + def _format_arguments(self, tool_args: Dict[str, Any]) -> List[str]: + """Format arguments for command-line execution. + + Converts a dictionary of arguments into command-line flags and values. + + Args: + tool_args: Dictionary of argument names and values + + Returns: + List of command-line arguments + """ + args = [] + for key, value in tool_args.items(): + if isinstance(value, bool): + if value: + args.append(f"--{key}") + elif isinstance(value, (list, tuple)): + for item in value: + args.extend([f"--{key}", str(item)]) + else: + args.extend([f"--{key}", str(value)]) + return args + + def _extract_utcp_manual_from_output(self, output: str, provider_name: str) -> Optional[UtcpManual]: + """Extract a UTCP manual from command output. + + Tries to parse the output as a UTCP manual. If it instead looks like a list of tools, + wraps them in a basic UtcpManual structure. + """ + # Try to parse the entire output as JSON first + try: + data = json.loads(output.strip()) + if isinstance(data, dict) and "utcp_version" in data and "tools" in data: + try: + return UtcpManualSerializer().validate_dict(data) + except Exception as e: + self._log_error( + f"Invalid UTCP manual format from provider '{provider_name}': {e}" + ) + # Fallback: try to parse tools from possibly-legacy structure + tools = self._parse_tool_data(data, provider_name) + if tools: + return UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=tools) + return None + # Fallback: try to parse as tools + tools = self._parse_tool_data(data, provider_name) + if tools: + return UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=tools) + except json.JSONDecodeError: + pass + + # Look for JSON objects within the output text and aggregate tools + aggregated_tools: List[Tool] = [] + lines = output.split('\n') + for line in lines: + line = line.strip() + if line.startswith('{') and line.endswith('}'): + try: + data = json.loads(line) + # If a full manual is found in a line, return it immediately + if isinstance(data, dict) and "utcp_version" in data and "tools" in data: + try: + return UtcpManualSerializer().validate_dict(data) + except Exception as e: + self._log_error( + f"Invalid UTCP manual format from provider '{provider_name}': {e}" + ) + # Fallback: try to parse tools from possibly-legacy structure + tools = self._parse_tool_data(data, provider_name) + if tools: + return UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=tools) + return None + found_tools = self._parse_tool_data(data, provider_name) + aggregated_tools.extend(found_tools) + except json.JSONDecodeError: + continue + + if aggregated_tools: + return UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=aggregated_tools) + + return None + + def _build_tool_from_dict(self, tool_data: Any, provider_name: str) -> Optional[Tool]: + """Build a Tool object from a dictionary, supporting legacy keys. + + This maps legacy 'tool_provider' into the new 'tool_call_template' + using the appropriate call template serializers. + """ + try: + if isinstance(tool_data, dict): + # If already new-style and call template is a dict, validate it + if "tool_call_template" in tool_data and isinstance(tool_data["tool_call_template"], dict): + td = dict(tool_data) + td["tool_call_template"] = CallTemplateSerializer().validate_dict(td["tool_call_template"]) + return Tool(**td) + + # Legacy style: 'tool_provider' + if "tool_provider" in tool_data and isinstance(tool_data["tool_provider"], dict): + provider = tool_data["tool_provider"] + provider_type = provider.get("provider_type") or provider.get("type") + # Normalize to call template dict + call_template_dict = {k: v for k, v in provider.items() if k != "provider_type"} + call_template_dict["type"] = provider_type + + # Validate based on type + if provider_type == "cli": + call_template = CliCallTemplateSerializer().validate_dict(call_template_dict) + else: + call_template = CallTemplateSerializer().validate_dict(call_template_dict) + + td = dict(tool_data) + td.pop("tool_provider", None) + td["tool_call_template"] = call_template + return Tool(**td) + + # Already a Tool-like dict with correct fields + return Tool(**tool_data) + except Exception as e: + self._log_error(f"Invalid tool definition from provider '{provider_name}': {e}") + return None + return None + + def _parse_tool_data(self, data: Any, provider_name: str) -> List[Tool]: + """Parse tool data from JSON. + + Supports both the new format (with 'tool_call_template') and the + legacy format (with 'tool_provider'). + + Args: + data: JSON data to parse + provider_name: Name of the provider for logging + + Returns: + List of tools parsed from the data + """ + tools: List[Tool] = [] + if isinstance(data, dict): + if 'tools' in data and isinstance(data['tools'], list): + for item in data['tools']: + built = self._build_tool_from_dict(item, provider_name) + if built is not None: + tools.append(built) + return tools + elif 'name' in data and 'description' in data: + built = self._build_tool_from_dict(data, provider_name) + return [built] if built is not None else [] + elif isinstance(data, list): + for item in data: + built = self._build_tool_from_dict(item, provider_name) + if built is not None: + tools.append(built) + return tools + + return tools + + async def call_tool(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + """Call a CLI tool. + + Executes the command specified by provider.command_name with the provided arguments. + + Args: + caller: The UTCP client that is calling this method. + tool_name: Name of the tool to call + tool_args: Arguments for the tool call + tool_call_template: The CliCallTemplate for the tool + + Returns: + The output from the command execution based on exit code: + - If exit code is 0: stdout (parsed as JSON if possible, otherwise raw string) + - If exit code is not 0: stderr + + Raises: + ValueError: If provider is not a CliProvider or command_name is not set + """ + if not isinstance(tool_call_template, CliCallTemplate): + raise ValueError("CliCommunicationProtocol can only be used with CliCallTemplate") + + if not tool_call_template.command_name: + raise ValueError(f"CliCallTemplate '{tool_call_template.name}' must have command_name set") + + # Build the command + # Parse command string into proper arguments + # Use posix=False on Windows, posix=True on Unix-like systems + command = shlex.split(tool_call_template.command_name, posix=(os.name != 'nt')) + + # Add formatted arguments + if tool_args: + command.extend(self._format_arguments(tool_args)) + + self._log_info(f"Executing CLI tool '{tool_name}': {' '.join(command)}") + + try: + env = self._prepare_environment(tool_call_template) + + stdout, stderr, return_code = await self._execute_command( + command, + env, + timeout=60.0, # Longer timeout for tool execution + working_dir=tool_call_template.working_dir + ) + + # Get output based on exit code + if return_code == 0: + output = stdout + self._log_info(f"CLI tool '{tool_name}' executed successfully (exit code 0)") + else: + output = stderr + self._log_info(f"CLI tool '{tool_name}' exited with code {return_code}, returning stderr") + + # Try to parse output as JSON, fall back to raw string + if output.strip(): + try: + result = json.loads(output) + self._log_info(f"Returning JSON output from CLI tool '{tool_name}'") + return result + except json.JSONDecodeError: + # Return raw string output + self._log_info(f"Returning text output from CLI tool '{tool_name}'") + return output.strip() + else: + self._log_info(f"CLI tool '{tool_name}' produced no output") + return "" + + except Exception as e: + self._log_error(f"Error executing CLI tool '{tool_name}': {e}") + raise + + async def call_tool_streaming(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> AsyncGenerator[Any, None]: + """Streaming calls are not supported for CLI protocol.""" + raise NotImplementedError("Streaming is not supported by the CLI communication protocol.") + + async def close(self) -> None: + """Close the transport. + + This is a no-op for CLI transports since they don't maintain connections. + """ + self._log_info("Closing CLI transport (no-op)") diff --git a/tests/client/transport_interfaces/test_cli_transport.py b/plugins/communication_protocols/cli/tests/test_cli_communication_protocol.py similarity index 74% rename from tests/client/transport_interfaces/test_cli_transport.py rename to plugins/communication_protocols/cli/tests/test_cli_communication_protocol.py index 9adf4dd..61b665c 100644 --- a/tests/client/transport_interfaces/test_cli_transport.py +++ b/plugins/communication_protocols/cli/tests/test_cli_communication_protocol.py @@ -12,16 +12,18 @@ import pytest import pytest_asyncio -from utcp.client.transport_interfaces.cli_transport import CliTransport -from utcp.shared.provider import CliProvider +from utcp_cli.cli_communication_protocol import CliCommunicationProtocol +from utcp_cli.cli_call_template import CliCallTemplate @pytest_asyncio.fixture -async def transport() -> CliTransport: - """Provides a clean CliTransport instance.""" - t = CliTransport() +async def transport() -> CliCommunicationProtocol: + """Provides a clean CliCommunicationProtocol instance.""" + t = CliCommunicationProtocol() yield t - await t.close() + # Optional cleanup if close() exists + if hasattr(t, "close") and asyncio.iscoroutinefunction(getattr(t, "close")): + await t.close() @pytest_asyncio.fixture @@ -187,15 +189,16 @@ def python_executable(): @pytest.mark.asyncio -async def test_register_provider_discovers_tools(transport: CliTransport, mock_cli_script, python_executable): +async def test_register_provider_discovers_tools(transport: CliCommunicationProtocol, mock_cli_script, python_executable): """Test that registering a provider discovers tools from command output.""" - provider = CliProvider( - name="mock_cli_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {mock_cli_script}" ) - tools = await transport.register_tool_provider(provider) + result = await transport.register_manual(None, call_template) + assert result is not None and result.manual is not None + tools = result.manual.tools assert len(tools) == 2 assert tools[0].name == "echo" assert tools[0].description == "Echo back the input" @@ -207,69 +210,62 @@ async def test_register_provider_discovers_tools(transport: CliTransport, mock_c @pytest.mark.asyncio -async def test_register_provider_missing_command_name(transport: CliTransport): +async def test_register_provider_missing_command_name(transport: CliCommunicationProtocol): """Test that registering a provider with empty command_name raises an error.""" - provider = CliProvider( - name="missing_command_provider", + call_template = CliCallTemplate( command_name="" # Empty string instead of missing field ) - with pytest.raises(ValueError, match="must have command_name set"): - await transport.register_tool_provider(provider) + with pytest.raises(ValueError): + await transport.register_manual(None, call_template) @pytest.mark.asyncio -async def test_register_provider_wrong_type(transport: CliTransport): - """Test that registering a non-CLI provider raises an error.""" - from utcp.shared.provider import HttpProvider +async def test_register_provider_wrong_type(transport: CliCommunicationProtocol): + """Test that registering a non-CLI call template raises an error.""" + class DummyTemplate: + type = "http" + command_name = "echo" - provider = HttpProvider( - name="http_provider", - url="https://example.com" - ) - - with pytest.raises(ValueError, match="CliTransport can only be used with CliProvider"): - await transport.register_tool_provider(provider) + with pytest.raises(ValueError): + await transport.register_manual(None, DummyTemplate()) @pytest.mark.asyncio -async def test_call_tool_json_output(transport: CliTransport, mock_cli_script, python_executable): +async def test_call_tool_json_output(transport: CliCommunicationProtocol, mock_cli_script, python_executable): """Test calling a tool that returns JSON output.""" - provider = CliProvider( - name="mock_cli_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {mock_cli_script}" ) - result = await transport.call_tool("echo", {"message": "Hello World"}, provider) + result = await transport.call_tool(None, "echo", {"message": "Hello World"}, call_template) assert isinstance(result, dict) assert result["result"] == "Echo: Hello World" @pytest.mark.asyncio -async def test_call_tool_math_operation(transport: CliTransport, mock_cli_script, python_executable): +async def test_call_tool_math_operation(transport: CliCommunicationProtocol, mock_cli_script, python_executable): """Test calling a math tool with numeric arguments.""" - provider = CliProvider( - name="mock_cli_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {mock_cli_script}" ) - result = await transport.call_tool("math", {"operation": "add", "a": 5, "b": 3}, provider) + result = await transport.call_tool(None, "math", {"operation": "add", "a": 5, "b": 3}, call_template) assert isinstance(result, dict) assert result["result"] == 8 @pytest.mark.asyncio -async def test_call_tool_error_handling(transport: CliTransport, mock_cli_script, python_executable): +async def test_call_tool_error_handling(transport: CliCommunicationProtocol, mock_cli_script, python_executable): """Test calling a tool that exits with an error returns stderr.""" - provider = CliProvider( - name="mock_cli_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {mock_cli_script}" ) # This should trigger an error in the mock script - result = await transport.call_tool("error_tool", {"error": "test error"}, provider) + result = await transport.call_tool(None, "error_tool", {"error": "test error"}, call_template) # Should return stderr content since exit code != 0 assert isinstance(result, str) @@ -277,33 +273,29 @@ async def test_call_tool_error_handling(transport: CliTransport, mock_cli_script @pytest.mark.asyncio -async def test_call_tool_missing_command_name(transport: CliTransport): +async def test_call_tool_missing_command_name(transport: CliCommunicationProtocol): """Test calling a tool with empty command_name raises an error.""" - provider = CliProvider( - name="missing_command_provider", + call_template = CliCallTemplate( command_name="" # Empty string instead of missing field ) - with pytest.raises(ValueError, match="must have command_name set"): - await transport.call_tool("some_tool", {}, provider) + with pytest.raises(ValueError): + await transport.call_tool(None, "some_tool", {}, call_template) @pytest.mark.asyncio -async def test_call_tool_wrong_provider_type(transport: CliTransport): +async def test_call_tool_wrong_provider_type(transport: CliCommunicationProtocol): """Test calling a tool with wrong provider type.""" - from utcp.shared.provider import HttpProvider - - provider = HttpProvider( - name="http_provider", - url="https://example.com" - ) + class DummyTemplate: + type = "http" + command_name = "echo" - with pytest.raises(ValueError, match="CliTransport can only be used with CliProvider"): - await transport.call_tool("some_tool", {}, provider) + with pytest.raises(ValueError): + await transport.call_tool(None, "some_tool", {}, DummyTemplate()) @pytest.mark.asyncio -async def test_environment_variables(transport: CliTransport, mock_cli_script, python_executable): +async def test_environment_variables(transport: CliCommunicationProtocol, mock_cli_script, python_executable): """Test that custom environment variables are properly set.""" env_vars = { "MY_API_KEY": "test-api-key-123", @@ -311,14 +303,13 @@ async def test_environment_variables(transport: CliTransport, mock_cli_script, p "CUSTOM_CONFIG": "config-data" } - provider = CliProvider( - name="env_cli_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {mock_cli_script}", env_vars=env_vars ) # Call the env check endpoint - result = await transport.call_tool("check_env", {"check-env": True}, provider) + result = await transport.call_tool(None, "check_env", {"check-env": True}, call_template) assert isinstance(result, dict) assert result["MY_API_KEY"] == "test-api-key-123" @@ -327,16 +318,15 @@ async def test_environment_variables(transport: CliTransport, mock_cli_script, p @pytest.mark.asyncio -async def test_no_environment_variables(transport: CliTransport, mock_cli_script, python_executable): +async def test_no_environment_variables(transport: CliCommunicationProtocol, mock_cli_script, python_executable): """Test that no environment variables are set when env_vars is None.""" - provider = CliProvider( - name="no_env_cli_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {mock_cli_script}" # env_vars=None by default ) # Call the env check endpoint - result = await transport.call_tool("check_env", {"check-env": True}, provider) + result = await transport.call_tool(None, "check_env", {"check-env": True}, call_template) assert isinstance(result, dict) # Should be empty since no custom env vars were set @@ -344,7 +334,7 @@ async def test_no_environment_variables(transport: CliTransport, mock_cli_script @pytest.mark.asyncio -async def test_working_directory(transport: CliTransport, mock_cli_script, python_executable, tmp_path): +async def test_working_directory(transport: CliCommunicationProtocol, mock_cli_script, python_executable, tmp_path): """Test that working directory is properly set during command execution.""" # Create a test file in a specific directory test_dir = tmp_path / "test_working_dir" @@ -367,14 +357,13 @@ async def test_working_directory(transport: CliTransport, mock_cli_script, pytho working_dir_script = tmp_path / "working_dir_script.py" working_dir_script.write_text(script_content) - provider = CliProvider( - name="working_dir_test_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {working_dir_script}", working_dir=str(test_dir) ) # Call the tool which should write the current directory to a file - result = await transport.call_tool("write_cwd", {"write-cwd": True}, provider) + result = await transport.call_tool(None, "write_cwd", {"write-cwd": True}, call_template) # Verify the result assert isinstance(result, dict) @@ -389,23 +378,22 @@ async def test_working_directory(transport: CliTransport, mock_cli_script, pytho @pytest.mark.asyncio -async def test_no_working_directory(transport: CliTransport, mock_cli_script, python_executable): +async def test_no_working_directory(transport: CliCommunicationProtocol, mock_cli_script, python_executable): """Test that commands work normally when no working directory is specified.""" - provider = CliProvider( - name="no_working_dir_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {mock_cli_script}" # working_dir=None by default ) # This should work normally - calling the echo tool - result = await transport.call_tool("echo", {"message": "test"}, provider) + result = await transport.call_tool(None, "echo", {"message": "test"}, call_template) assert isinstance(result, dict) assert result["result"] == "Echo: test" @pytest.mark.asyncio -async def test_env_vars_and_working_dir_combined(transport: CliTransport, python_executable, tmp_path): +async def test_env_vars_and_working_dir_combined(transport: CliCommunicationProtocol, python_executable, tmp_path): """Test that both environment variables and working directory work together.""" # Create a test directory test_dir = tmp_path / "combined_test_dir" @@ -431,15 +419,14 @@ async def test_env_vars_and_working_dir_combined(transport: CliTransport, python combined_script = tmp_path / "combined_test_script.py" combined_script.write_text(script_content) - provider = CliProvider( - name="combined_test_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {combined_script}", env_vars={"TEST_COMBINED_VAR": "test_value_123"}, working_dir=str(test_dir) ) # Call the tool - result = await transport.call_tool("combined_test", {"combined-test": True}, provider) + result = await transport.call_tool(None, "combined_test", {"combined-test": True}, call_template) # Verify both environment variable and working directory are set correctly assert isinstance(result, dict) @@ -451,7 +438,7 @@ async def test_env_vars_and_working_dir_combined(transport: CliTransport, python @pytest.mark.asyncio async def test_argument_formatting(): """Test that arguments are properly formatted for command line.""" - transport = CliTransport() + transport = CliCommunicationProtocol() # Test various argument types args = { @@ -482,13 +469,14 @@ async def test_argument_formatting(): @pytest.mark.asyncio async def test_json_extraction_from_output(): """Test extracting JSON from various output formats.""" - transport = CliTransport() + transport = CliCommunicationProtocol() # Test complete JSON output output1 = '{"tools": [{"name": "test", "description": "Test tool", "tool_provider": {"provider_type": "cli", "name": "test_provider", "command_name": "test"}}]}' - tools1 = transport._extract_utcp_manual_from_output(output1, "test_provider") - assert len(tools1) == 1 - assert tools1[0].name == "test" + manual1 = transport._extract_utcp_manual_from_output(output1, "test_provider") + assert manual1 is not None + assert len(manual1.tools) == 1 + assert manual1.tools[0].name == "test" # Test JSON within text output output2 = ''' @@ -496,46 +484,48 @@ async def test_json_extraction_from_output(): {"tools": [{"name": "embedded", "description": "Embedded tool", "tool_provider": {"provider_type": "cli", "name": "test_provider", "command_name": "test"}}]} Process completed. ''' - tools2 = transport._extract_utcp_manual_from_output(output2, "test_provider") - assert len(tools2) == 1 - assert tools2[0].name == "embedded" + manual2 = transport._extract_utcp_manual_from_output(output2, "test_provider") + assert manual2 is not None + assert len(manual2.tools) == 1 + assert manual2.tools[0].name == "embedded" # Test single tool definition output3 = '{"name": "single", "description": "Single tool", "tool_provider": {"provider_type": "cli", "name": "test_provider", "command_name": "test"}}' - tools3 = transport._extract_utcp_manual_from_output(output3, "test_provider") - assert len(tools3) == 1 - assert tools3[0].name == "single" + manual3 = transport._extract_utcp_manual_from_output(output3, "test_provider") + assert manual3 is not None + assert len(manual3.tools) == 1 + assert manual3.tools[0].name == "single" # Test no valid JSON output4 = "No JSON here, just plain text" - tools4 = transport._extract_utcp_manual_from_output(output4, "test_provider") - assert len(tools4) == 0 + manual4 = transport._extract_utcp_manual_from_output(output4, "test_provider") + assert manual4 is None @pytest.mark.asyncio -async def test_deregister_provider(transport: CliTransport, mock_cli_script, python_executable): +async def test_deregister_provider(transport: CliCommunicationProtocol, mock_cli_script, python_executable): """Test deregistering a CLI provider.""" - provider = CliProvider( - name="mock_cli_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {mock_cli_script}" ) # Register and then deregister (should not raise any errors) - await transport.register_tool_provider(provider) - await transport.deregister_tool_provider(provider) + await transport.register_manual(None, call_template) + await transport.deregister_manual(None, call_template) @pytest.mark.asyncio -async def test_close_transport(transport: CliTransport): +async def test_close_transport(transport: CliCommunicationProtocol): """Test closing the transport.""" - # Should not raise any errors - await transport.close() + # Should not raise any errors (only if close() is implemented) + if hasattr(transport, "close") and asyncio.iscoroutinefunction(getattr(transport, "close")): + await transport.close() @pytest.mark.asyncio async def test_command_execution_timeout(python_executable, tmp_path): """Test that command execution respects timeout.""" - transport = CliTransport() + transport = CliCommunicationProtocol() # Create a Python script that sleeps for a long time sleep_script_content = ''' @@ -567,7 +557,7 @@ async def test_command_execution_timeout(python_executable, tmp_path): @pytest.mark.asyncio -async def test_mixed_output_formats(transport: CliTransport, python_executable): +async def test_mixed_output_formats(transport: CliCommunicationProtocol, python_executable): """Test handling of mixed output formats (text and JSON).""" # Create a simple script that outputs mixed content script_content = ''' @@ -582,12 +572,11 @@ async def test_mixed_output_formats(transport: CliTransport, python_executable): script_path = f.name try: - provider = CliProvider( - name="mixed_output_provider", + call_template = CliCallTemplate( command_name=f"{python_executable} {script_path}" ) - result = await transport.call_tool("mixed_tool", {}, provider) + result = await transport.call_tool(None, "mixed_tool", {}, call_template) # Should return the JSON part since command succeeds (exit code 0) # But the output contains both text and JSON diff --git a/plugins/communication_protocols/gql/INCOMPLETE b/plugins/communication_protocols/gql/INCOMPLETE new file mode 100644 index 0000000..e69de29 diff --git a/plugins/communication_protocols/gql/old_tests/test_graphql_transport.py b/plugins/communication_protocols/gql/old_tests/test_graphql_transport.py new file mode 100644 index 0000000..d33c323 --- /dev/null +++ b/plugins/communication_protocols/gql/old_tests/test_graphql_transport.py @@ -0,0 +1,129 @@ +# import pytest +# import pytest_asyncio +# import json +# from aiohttp import web +# from utcp.client.transport_interfaces.graphql_transport import GraphQLClientTransport +# from utcp.shared.provider import GraphQLProvider +# from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth + + +# @pytest_asyncio.fixture +# async def graphql_app(): +# async def graphql_handler(request): +# body = await request.json() +# query = body.get("query", "") +# variables = body.get("variables", {}) +# # Introspection query (minimal response) +# if "__schema" in query: +# return web.json_response({ +# "data": { +# "__schema": { +# "queryType": {"name": "Query"}, +# "mutationType": {"name": "Mutation"}, +# "subscriptionType": None, +# "types": [ +# {"kind": "OBJECT", "name": "Query", "fields": [ +# {"name": "hello", "args": [{"name": "name", "type": {"kind": "SCALAR", "name": "String"}, "defaultValue": None}], "type": {"kind": "SCALAR", "name": "String"}, "isDeprecated": False, "deprecationReason": None} +# ], "interfaces": []}, +# {"kind": "OBJECT", "name": "Mutation", "fields": [ +# {"name": "add", "args": [ +# {"name": "a", "type": {"kind": "SCALAR", "name": "Int"}, "defaultValue": None}, +# {"name": "b", "type": {"kind": "SCALAR", "name": "Int"}, "defaultValue": None} +# ], "type": {"kind": "SCALAR", "name": "Int"}, "isDeprecated": False, "deprecationReason": None} +# ], "interfaces": []}, +# {"kind": "SCALAR", "name": "String"}, +# {"kind": "SCALAR", "name": "Int"}, +# {"kind": "SCALAR", "name": "Boolean"} +# ], +# "directives": [] +# } +# } +# }) +# # hello query +# if "hello" in query: +# name = variables.get("name", "world") +# return web.json_response({"data": {"hello": f"Hello, {name}!"}}) +# # add mutation +# if "add" in query: +# a = variables.get("a", 0) +# b = variables.get("b", 0) +# return web.json_response({"data": {"add": a + b}}) +# # fallback +# return web.json_response({"data": {}}, status=200) + +# app = web.Application() +# app.router.add_post("/graphql", graphql_handler) +# return app + +# @pytest_asyncio.fixture +# async def aiohttp_graphql_client(aiohttp_client, graphql_app): +# return await aiohttp_client(graphql_app) + +# @pytest_asyncio.fixture +# def transport(): +# return GraphQLClientTransport() + +# @pytest_asyncio.fixture +# def provider(aiohttp_graphql_client): +# return GraphQLProvider( +# name="test-graphql-provider", +# url=str(aiohttp_graphql_client.make_url("/graphql")), +# headers={}, +# ) + +# @pytest.mark.asyncio +# async def test_register_tool_provider_discovers_tools(transport, provider): +# tools = await transport.register_tool_provider(provider) +# tool_names = [tool.name for tool in tools] +# assert "hello" in tool_names +# assert "add" in tool_names + +# @pytest.mark.asyncio +# async def test_call_tool_query(transport, provider): +# result = await transport.call_tool("hello", {"name": "Alice"}, provider) +# assert result["hello"] == "Hello, Alice!" + +# @pytest.mark.asyncio +# async def test_call_tool_mutation(transport, provider): +# provider.operation_type = "mutation" +# mutation = ''' +# mutation ($a: Int, $b: Int) { +# add(a: $a, b: $b) +# } +# ''' +# result = await transport.call_tool("add", {"a": 2, "b": 3}, provider, query=mutation) +# assert result["add"] == 5 + +# @pytest.mark.asyncio +# async def test_call_tool_api_key(transport, provider): +# provider.headers = {} +# provider.auth = ApiKeyAuth(var_name="X-API-Key", api_key="test-key") +# result = await transport.call_tool("hello", {"name": "Bob"}, provider) +# assert result["hello"] == "Hello, Bob!" + +# @pytest.mark.asyncio +# async def test_call_tool_basic_auth(transport, provider): +# provider.headers = {} +# provider.auth = BasicAuth(username="user", password="pass") +# result = await transport.call_tool("hello", {"name": "Eve"}, provider) +# assert result["hello"] == "Hello, Eve!" + +# @pytest.mark.asyncio +# async def test_call_tool_oauth2(monkeypatch, transport, provider): +# async def fake_oauth2(auth): +# return "fake-token" +# transport._handle_oauth2 = fake_oauth2 +# provider.headers = {} +# provider.auth = OAuth2Auth(token_url="http://fake/token", client_id="id", client_secret="secret", scope="scope") +# result = await transport.call_tool("hello", {"name": "Zoe"}, provider) +# assert result["hello"] == "Hello, Zoe!" + +# @pytest.mark.asyncio +# async def test_enforce_https_or_localhost_raises(transport, provider): +# provider.url = "http://evil.com/graphql" +# with pytest.raises(ValueError): +# await transport.call_tool("hello", {"name": "Mallory"}, provider) + +# @pytest.mark.asyncio +# async def test_deregister_tool_provider_noop(transport, provider): +# await transport.deregister_tool_provider(provider) \ No newline at end of file diff --git a/plugins/communication_protocols/gql/pyproject.toml b/plugins/communication_protocols/gql/pyproject.toml new file mode 100644 index 0000000..070e945 --- /dev/null +++ b/plugins/communication_protocols/gql/pyproject.toml @@ -0,0 +1,40 @@ +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" + +[project] +name = "utcp-gql" +version = "1.0.0" +authors = [ + { name = "UTCP Contributors" }, +] +description = "Universal Tool Calling Protocol (UTCP) client library for Python" +readme = "README.md" +requires-python = ">=3.10" +dependencies = [ + "pydantic>=2.0", + "gql>=3.0", + "utcp>=1.0" +] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "Programming Language :: Python :: 3", + "Operating System :: OS Independent", +] +license = "MPL-2.0" + +[project.optional-dependencies] +dev = [ + "build", + "pytest", + "pytest-asyncio", + "pytest-cov", + "coverage", + "twine", +] + +[project.urls] +Homepage = "https://utcp.io" +Source = "https://github.com/universal-tool-calling-protocol/python-utcp" +Issues = "https://github.com/universal-tool-calling-protocol/python-utcp/issues" \ No newline at end of file diff --git a/plugins/communication_protocols/gql/src/utcp_gql/__init__.py b/plugins/communication_protocols/gql/src/utcp_gql/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/plugins/communication_protocols/gql/src/utcp_gql/gql_call_template.py b/plugins/communication_protocols/gql/src/utcp_gql/gql_call_template.py new file mode 100644 index 0000000..dfe5b07 --- /dev/null +++ b/plugins/communication_protocols/gql/src/utcp_gql/gql_call_template.py @@ -0,0 +1,29 @@ +from utcp.data.call_template import CallTemplate +from utcp.data.auth import Auth +from typing import Dict, List, Optional, Literal +from pydantic import Field + +class GraphQLProvider(CallTemplate): + """Provider configuration for GraphQL-based tools. + + Enables communication with GraphQL endpoints supporting queries, mutations, + and subscriptions. Provides flexible query execution with custom headers + and authentication. + + Attributes: + call_template_type: Always "graphql" for GraphQL providers. + url: The GraphQL endpoint URL. + operation_type: The type of GraphQL operation (query, mutation, subscription). + operation_name: Optional name for the GraphQL operation. + auth: Optional authentication configuration. + headers: Optional static headers to include in requests. + header_fields: List of tool argument names to map to HTTP request headers. + """ + + call_template_type: Literal["graphql"] = "graphql" + url: str + operation_type: Literal["query", "mutation", "subscription"] = "query" + operation_name: Optional[str] = None + auth: Optional[Auth] = None + headers: Optional[Dict[str, str]] = None + header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers for the initial connection.") diff --git a/src/utcp/client/transport_interfaces/graphql_transport.py b/plugins/communication_protocols/gql/src/utcp_gql/gql_communication_protocol.py similarity index 93% rename from src/utcp/client/transport_interfaces/graphql_transport.py rename to plugins/communication_protocols/gql/src/utcp_gql/gql_communication_protocol.py index 6ad8053..523a97c 100644 --- a/src/utcp/client/transport_interfaces/graphql_transport.py +++ b/plugins/communication_protocols/gql/src/utcp_gql/gql_communication_protocol.py @@ -2,21 +2,22 @@ import aiohttp import asyncio import ssl -import logging from gql import Client as GqlClient, gql as gql_query from gql.transport.aiohttp import AIOHTTPTransport from utcp.client.client_transport_interface import ClientTransportInterface from utcp.shared.provider import Provider, GraphQLProvider from utcp.shared.tool import Tool, ToolInputOutputSchema from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth +import logging + +logger = logging.getLogger(__name__) class GraphQLClientTransport(ClientTransportInterface): """ Simple, robust, production-ready GraphQL transport using gql. Stateless, per-operation. Supports all GraphQL features. """ - def __init__(self, logger: Optional[Callable[[str, Any], None]] = None): - self._log = logger or (lambda msg, error=False: None) + def __init__(self): self._oauth_tokens: Dict[str, Dict[str, Any]] = {} def _enforce_https_or_localhost(self, url: str): @@ -102,7 +103,7 @@ async def deregister_tool_provider(self, manual_provider: Provider) -> None: # Stateless: nothing to do pass - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider, query: Optional[str] = None) -> Any: + async def call_tool(self, tool_name: str, tool_args: Dict[str, Any], tool_provider: Provider, query: Optional[str] = None) -> Any: if not isinstance(tool_provider, GraphQLProvider): raise ValueError("GraphQLClientTransport can only be used with GraphQLProvider") self._enforce_https_or_localhost(tool_provider.url) @@ -111,19 +112,19 @@ async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provid async with GqlClient(transport=transport, fetch_schema_from_transport=True) as session: if query is not None: document = gql_query(query) - result = await session.execute(document, variable_values=arguments) + result = await session.execute(document, variable_values=tool_args) return result # If no query provided, build a simple query # Default to query operation op_type = getattr(tool_provider, 'operation_type', 'query') - arg_str = ', '.join(f"${k}: String" for k in arguments.keys()) + arg_str = ', '.join(f"${k}: String" for k in tool_args.keys()) var_defs = f"({arg_str})" if arg_str else "" - arg_pass = ', '.join(f"{k}: ${k}" for k in arguments.keys()) + arg_pass = ', '.join(f"{k}: ${k}" for k in tool_args.keys()) arg_pass = f"({arg_pass})" if arg_pass else "" gql_str = f"{op_type} {var_defs} {{ {tool_name}{arg_pass} }}" document = gql_query(gql_str) - result = await session.execute(document, variable_values=arguments) + result = await session.execute(document, variable_values=tool_args) return result async def close(self) -> None: - self._oauth_tokens.clear() \ No newline at end of file + self._oauth_tokens.clear() diff --git a/pyproject.toml b/plugins/communication_protocols/http/pyproject.toml similarity index 75% rename from pyproject.toml rename to plugins/communication_protocols/http/pyproject.toml index 9f1f5d6..97f332f 100644 --- a/pyproject.toml +++ b/plugins/communication_protocols/http/pyproject.toml @@ -3,14 +3,10 @@ requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" [project] -name = "utcp" -version = "0.2.3" +name = "utcp-http" +version = "1.0.0" authors = [ - { name = "Razvan-Ion Radulescu" }, - { name = "Andrei-Stefan Ghiurtu" }, - { name = "Juan Viera Garcia" }, - { name = "Ali Raza" }, - { name = "Ulugbek Isroilov" } + { name = "UTCP Contributors" }, ] description = "Universal Tool Calling Protocol (UTCP) client library for Python" readme = "README.md" @@ -18,12 +14,9 @@ requires-python = ">=3.10" dependencies = [ "pydantic>=2.0", "authlib>=1.0", - "python-dotenv>=1.0", - "tomli>=2.0", "aiohttp>=3.8", - "mcp>=1.0", "pyyaml>=6.0", - "gql>=3.0", + "utcp>=1.0" ] classifiers = [ "Development Status :: 4 - Beta", @@ -49,4 +42,7 @@ dev = [ [project.urls] Homepage = "https://utcp.io" Source = "https://github.com/universal-tool-calling-protocol/python-utcp" -Issues = "https://github.com/universal-tool-calling-protocol/python-utcp/issues" \ No newline at end of file +Issues = "https://github.com/universal-tool-calling-protocol/python-utcp/issues" + +[project.entry-points."utcp.plugins"] +http = "utcp_http:register" \ No newline at end of file diff --git a/plugins/communication_protocols/http/src/utcp_http/__init__.py b/plugins/communication_protocols/http/src/utcp_http/__init__.py new file mode 100644 index 0000000..cb1f7a0 --- /dev/null +++ b/plugins/communication_protocols/http/src/utcp_http/__init__.py @@ -0,0 +1,39 @@ +"""HTTP Communication Protocol plugin for UTCP. + +This plugin provides HTTP-based communication protocols including: +- Standard HTTP requests +- Server-Sent Events (SSE) +- Streamable HTTP with chunked transfer encoding +""" + +from utcp.plugins.discovery import register_communication_protocol, register_call_template +from utcp_http.http_communication_protocol import HttpCommunicationProtocol +from utcp_http.sse_communication_protocol import SseCommunicationProtocol +from utcp_http.streamable_http_communication_protocol import StreamableHttpCommunicationProtocol +from utcp_http.http_call_template import HttpCallTemplate, HttpCallTemplateSerializer +from utcp_http.sse_call_template import SseCallTemplate, SSECallTemplateSerializer +from utcp_http.streamable_http_call_template import StreamableHttpCallTemplate, StreamableHttpCallTemplateSerializer + +def register(): + # Register HTTP communication protocols + register_communication_protocol("http", HttpCommunicationProtocol()) + register_communication_protocol("sse", SseCommunicationProtocol()) + register_communication_protocol("streamable_http", StreamableHttpCommunicationProtocol()) + + # Register call template serializers + register_call_template("http", HttpCallTemplateSerializer()) + register_call_template("sse", SSECallTemplateSerializer()) + register_call_template("streamable_http", StreamableHttpCallTemplateSerializer()) + +# Export public API +__all__ = [ + "HttpCommunicationProtocol", + "SseCommunicationProtocol", + "StreamableHttpCommunicationProtocol", + "HttpCallTemplate", + "SseCallTemplate", + "StreamableHttpCallTemplate", + "HttpCallTemplateSerializer", + "SSECallTemplateSerializer", + "StreamableHttpCallTemplateSerializer", +] diff --git a/plugins/communication_protocols/http/src/utcp_http/http_call_template.py b/plugins/communication_protocols/http/src/utcp_http/http_call_template.py new file mode 100644 index 0000000..c422aeb --- /dev/null +++ b/plugins/communication_protocols/http/src/utcp_http/http_call_template.py @@ -0,0 +1,49 @@ +from utcp.data.call_template import CallTemplate, CallTemplateSerializer +from utcp.data.auth import Auth +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback +from typing import Optional, Dict, List, Literal +from pydantic import Field + +class HttpCallTemplate(CallTemplate): + """Provider configuration for HTTP-based tools. + + Supports RESTful HTTP/HTTPS APIs with various HTTP methods, authentication, + custom headers, and flexible request/response handling. Supports URL path + parameters using {parameter_name} syntax. All tool arguments not mapped to + URL body, headers or query pattern parameters are passed as query parameters using '?arg_name={arg_value}'. + + Attributes: + call_template_type: Always "http" for HTTP providers. + http_method: The HTTP method to use for requests. + url: The base URL for the HTTP endpoint. Supports path parameters like + "https://api.example.com/users/{user_id}/posts/{post_id}". + content_type: The Content-Type header for requests. + auth: Optional authentication configuration. + headers: Optional static headers to include in all requests. + body_field: Name of the tool argument to map to the HTTP request body. + header_fields: List of tool argument names to map to HTTP request headers. + """ + + call_template_type: Literal["http"] = "http" + http_method: Literal["GET", "POST", "PUT", "DELETE", "PATCH"] = "GET" + url: str + content_type: str = Field(default="application/json") + auth: Optional[Auth] = None + headers: Optional[Dict[str, str]] = None + body_field: Optional[str] = Field(default="body", description="The name of the single input field to be sent as the request body.") + header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers.") + + +class HttpCallTemplateSerializer(Serializer[HttpCallTemplate]): + """Serializer for HttpCallTemplate.""" + + def to_dict(self, obj: HttpCallTemplate) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> HttpCallTemplate: + try: + return HttpCallTemplate.model_validate(obj) + except Exception as e: + raise UtcpSerializerValidationError("Invalid HttpCallTemplate: " + traceback.format_exc()) from e diff --git a/src/utcp/client/transport_interfaces/http_transport.py b/plugins/communication_protocols/http/src/utcp_http/http_communication_protocol.py similarity index 57% rename from src/utcp/client/transport_interfaces/http_transport.py rename to plugins/communication_protocols/http/src/utcp_http/http_communication_protocol.py index 3aa295b..b0cf39f 100644 --- a/src/utcp/client/transport_interfaces/http_transport.py +++ b/plugins/communication_protocols/http/src/utcp_http/http_communication_protocol.py @@ -1,6 +1,6 @@ -"""HTTP transport implementation for UTCP client. +"""HTTP communication protocol implementation for UTCP client. -This module provides the HTTP transport implementation that handles communication +This module provides the HTTP communication protocol implementation that handles communication with HTTP-based tool providers. It supports RESTful APIs, authentication methods, URL path parameters, and automatic tool discovery through various formats. @@ -12,24 +12,31 @@ - Request/response handling with proper error management """ -from typing import Dict, Any, List +from typing import Dict, Any, List, Optional, Callable, AsyncGenerator import aiohttp import json import yaml import base64 import re - -from utcp.client.client_transport_interface import ClientTransportInterface -from utcp.shared.provider import Provider, HttpProvider -from utcp.shared.tool import Tool -from utcp.shared.utcp_manual import UtcpManual -from utcp.client.openapi_converter import OpenApiConverter -from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth -from typing import Optional, Callable +import traceback + +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.data.call_template import CallTemplate +from utcp.data.tool import Tool +from utcp.data.utcp_manual import UtcpManual, UtcpManualSerializer +from utcp.data.register_manual_response import RegisterManualResult +from utcp.data.auth_implementations.api_key_auth import ApiKeyAuth +from utcp.data.auth_implementations.basic_auth import BasicAuth +from utcp.data.auth_implementations.oauth2_auth import OAuth2Auth +from utcp_http.http_call_template import HttpCallTemplate from aiohttp import ClientSession, BasicAuth as AiohttpBasicAuth +from utcp_http.openapi_converter import OpenApiConverter +import logging + +logger = logging.getLogger(__name__) -class HttpClientTransport(ClientTransportInterface): - """HTTP transport implementation for UTCP client. +class HttpCommunicationProtocol(CommunicationProtocol): + """HTTP communication protocol implementation for UTCP client. Handles communication with HTTP-based tool providers, supporting various authentication methods, URL path parameters, and automatic tool discovery. @@ -59,10 +66,8 @@ def __init__(self, logger: Optional[Callable[[str], None]] = None): """ self._session: Optional[aiohttp.ClientSession] = None self._oauth_tokens: Dict[str, Dict[str, Any]] = {} - - self._log = logger or (lambda *args, **kwargs: None) - def _apply_auth(self, provider: HttpProvider, headers: Dict[str, str], query_params: Dict[str, Any]) -> tuple: + def _apply_auth(self, provider: HttpCallTemplate, headers: Dict[str, str], query_params: Dict[str, Any]) -> tuple: """Apply authentication to the request based on the provider's auth configuration. Returns: @@ -81,7 +86,7 @@ def _apply_auth(self, provider: HttpProvider, headers: Dict[str, str], query_par elif provider.auth.location == "cookie": cookies[provider.auth.var_name] = provider.auth.api_key else: - self._log("API key not found for ApiKeyAuth.", error=True) + logger.error("API key not found for ApiKeyAuth.") raise ValueError("API key for ApiKeyAuth not found.") elif isinstance(provider.auth, BasicAuth): @@ -94,20 +99,21 @@ def _apply_auth(self, provider: HttpProvider, headers: Dict[str, str], query_par return auth, cookies - async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: - """Discover tools from a REST API provider. + async def register_manual(self, caller, manual_call_template: CallTemplate) -> RegisterManualResult: + """Register a manual and its tools. Args: - provider: Details of the REST provider + caller: The UTCP client that is calling this method. + manual_call_template: The call template of the manual to register. Returns: - List of tool declarations as dictionaries, or None if discovery fails + RegisterManualResult object containing the call template and manual. """ - if not isinstance(manual_provider, HttpProvider): - raise ValueError("HttpTransport can only be used with HttpProvider") + if not isinstance(manual_call_template, HttpCallTemplate): + raise ValueError("HttpCommunicationProtocol can only be used with HttpCallTemplate") try: - url = manual_provider.url + url = manual_call_template.url # Security check: Enforce HTTPS or localhost to prevent MITM attacks if not (url.startswith("https://") or url.startswith("http://localhost") or url.startswith("http://127.0.0.1")): @@ -116,23 +122,23 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: "Non-secure URLs are vulnerable to man-in-the-middle attacks." ) - self._log(f"Discovering tools from '{manual_provider.name}' (REST) at {url}") + logger.info(f"Discovering tools from '{manual_call_template.name}' (HTTP) at {url}") - # Use the provider's configuration (headers, auth, HTTP method, etc.) - request_headers = manual_provider.headers.copy() if manual_provider.headers else {} + # Use the call template's configuration (headers, auth, HTTP method, etc.) + request_headers = manual_call_template.headers.copy() if manual_call_template.headers else {} body_content = None query_params = {} # Handle authentication - auth, cookies = self._apply_auth(manual_provider, request_headers, query_params) + auth, cookies = self._apply_auth(manual_call_template, request_headers, query_params) # Handle OAuth2 separately since it requires async token retrieval - if manual_provider.auth and isinstance(manual_provider.auth, OAuth2Auth): - token = await self._handle_oauth2(manual_provider.auth) + if manual_call_template.auth and isinstance(manual_call_template.auth, OAuth2Auth): + token = await self._handle_oauth2(manual_call_template.auth) request_headers["Authorization"] = f"Bearer {token}" # Handle body content if specified - if manual_provider.body_field: + if manual_call_template.body_field: # For discovery, we typically don't have body content, but support it if needed body_content = None @@ -140,7 +146,7 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: try: # Set content-type header if body is provided and header not already set if body_content is not None and "Content-Type" not in request_headers: - request_headers["Content-Type"] = manual_provider.content_type + request_headers["Content-Type"] = manual_call_template.content_type # Prepare body content based on content type data = None @@ -151,8 +157,8 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: else: data = body_content - # Make the request with the provider's HTTP method - method = manual_provider.http_method.lower() + # Make the request with the call template's HTTP method + method = manual_call_template.http_method.lower() request_method = getattr(session, method) async with request_method( @@ -177,67 +183,103 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: response_data = json.loads(response_text) # Check if the response is a UTCP manual or an OpenAPI spec - if "version" in response_data and "tools" in response_data: - self._log(f"Detected UTCP manual from '{manual_provider.name}'.") - utcp_manual = UtcpManual(**response_data) + if "utcp_version" in response_data and "tools" in response_data: + logger.info(f"Detected UTCP manual from '{manual_call_template.name}'.") + utcp_manual = UtcpManualSerializer().validate_dict(response_data) else: - self._log(f"Assuming OpenAPI spec from '{manual_provider.name}'. Converting to UTCP manual.") - converter = OpenApiConverter(response_data, spec_url=manual_provider.url, provider_name=manual_provider.name) + logger.info(f"Assuming OpenAPI spec from '{manual_call_template.name}'. Converting to UTCP manual.") + converter = OpenApiConverter(response_data, spec_url=manual_call_template.url, call_template_name=manual_call_template.name) utcp_manual = converter.convert() - return utcp_manual.tools + return RegisterManualResult( + success=True, + manual_call_template=manual_call_template, + manual=utcp_manual, + errors=[] + ) except aiohttp.ClientResponseError as e: - self._log(f"Error connecting to REST provider '{manual_provider.name}': {e}", error=True) - return [] + error_msg = f"Error connecting to HTTP provider '{manual_call_template.name}': {e}" + logger.error(error_msg) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[error_msg] + ) except (json.JSONDecodeError, yaml.YAMLError) as e: - self._log(f"Error parsing spec from REST provider '{manual_provider.name}': {e}", error=True) - return [] + error_msg = f"Error parsing spec from HTTP provider '{manual_call_template.name}': {e}" + logger.error(error_msg) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[error_msg] + ) except Exception as e: - self._log(f"Unexpected error discovering tools from REST provider '{manual_provider.name}': {e}", error=True) - return [] - - async def deregister_tool_provider(self, manual_provider: Provider) -> None: - """Deregistering a tool provider is a no-op for the stateless HTTP transport.""" + error_msg = f"Unexpected error discovering tools from HTTP provider '{manual_call_template.name}': {traceback.format_exc()}" + logger.error(error_msg) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[error_msg] + ) + + async def deregister_manual(self, caller, manual_call_template: CallTemplate) -> None: + """Deregister a manual and its tools. + + Deregistering a manual is a no-op for the stateless HTTP communication protocol. + """ pass - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> Any: - """Calls a tool on an HTTP provider.""" - if not isinstance(tool_provider, HttpProvider): - raise ValueError("HttpClientTransport can only be used with HttpProvider") + async def call_tool(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + """Execute a tool call through this transport. + + Args: + caller: The UTCP client that is calling this method. + tool_name: Name of the tool to call (may include provider prefix). + tool_args: Dictionary of arguments to pass to the tool. + tool_call_template: Call template of the tool to call. + + Returns: + The tool's response, with type depending on the tool's output schema. + """ + if not isinstance(tool_call_template, HttpCallTemplate): + raise ValueError("HttpCommunicationProtocol can only be used with HttpCallTemplate") - request_headers = tool_provider.headers.copy() if tool_provider.headers else {} + request_headers = tool_call_template.headers.copy() if tool_call_template.headers else {} body_content = None - remaining_args = arguments.copy() + remaining_args = tool_args.copy() # Handle header fields - if tool_provider.header_fields: - for field_name in tool_provider.header_fields: + if tool_call_template.header_fields: + for field_name in tool_call_template.header_fields: if field_name in remaining_args: request_headers[field_name] = str(remaining_args.pop(field_name)) # Handle body field - if tool_provider.body_field and tool_provider.body_field in remaining_args: - body_content = remaining_args.pop(tool_provider.body_field) + if tool_call_template.body_field and tool_call_template.body_field in remaining_args: + body_content = remaining_args.pop(tool_call_template.body_field) # Build the URL with path parameters substituted - url = self._build_url_with_path_params(tool_provider.url, remaining_args) + url = self._build_url_with_path_params(tool_call_template.url, remaining_args) # The rest of the arguments are query parameters query_params = remaining_args # Handle authentication - auth, cookies = self._apply_auth(tool_provider, request_headers, query_params) + auth, cookies = self._apply_auth(tool_call_template, request_headers, query_params) # Handle OAuth2 separately since it requires async token retrieval - if tool_provider.auth and isinstance(tool_provider.auth, OAuth2Auth): - token = await self._handle_oauth2(tool_provider.auth) + if tool_call_template.auth and isinstance(tool_call_template.auth, OAuth2Auth): + token = await self._handle_oauth2(tool_call_template.auth) request_headers["Authorization"] = f"Bearer {token}" async with aiohttp.ClientSession() as session: try: # Set content-type header if body is provided and header not already set if body_content is not None and "Content-Type" not in request_headers: - request_headers["Content-Type"] = tool_provider.content_type + request_headers["Content-Type"] = tool_call_template.content_type # Prepare body content based on content type data = None @@ -249,7 +291,7 @@ async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provid data = body_content # Make the request with the appropriate HTTP method - method = tool_provider.http_method.lower() + method = tool_call_template.http_method.lower() request_method = getattr(session, method) async with request_method( @@ -266,12 +308,28 @@ async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provid return await response.json() except aiohttp.ClientResponseError as e: - self._log(f"Error calling tool '{tool_name}' on provider '{tool_provider.name}': {e}", error=True) + logger.error(f"Error calling tool '{tool_name}' on call template '{tool_call_template.name}': {e}") raise except Exception as e: - self._log(f"Unexpected error calling tool '{tool_name}': {e}", error=True) + logger.error(f"Unexpected error calling tool '{tool_name}': {e}") raise + async def call_tool_streaming(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> AsyncGenerator[Any, None]: + """Execute a tool call through this transport streamingly. + + Args: + caller: The UTCP client that is calling this method. + tool_name: Name of the tool to call (may include provider prefix). + tool_args: Dictionary of arguments to pass to the tool. + tool_call_template: Call template of the tool to call. + + Returns: + An async generator that yields the tool's response. + """ + # For HTTP, streaming is not typically supported, so we'll just yield the complete response + result = await self.call_tool(caller, tool_name, tool_args, tool_call_template) + yield result + async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: """Handles OAuth2 client credentials flow, trying both body and auth header methods.""" client_id = auth_details.client_id @@ -282,7 +340,7 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: async with aiohttp.ClientSession() as session: # Method 1: Send credentials in the request body try: - self._log("Attempting OAuth2 token fetch with credentials in body.") + logger.info("Attempting OAuth2 token fetch with credentials in body.") body_data = { 'grant_type': 'client_credentials', 'client_id': auth_details.client_id, @@ -295,11 +353,11 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: self._oauth_tokens[client_id] = token_response return token_response["access_token"] except aiohttp.ClientError as e: - self._log(f"OAuth2 with credentials in body failed: {e}. Trying Basic Auth header.") + logger.error(f"OAuth2 with credentials in body failed: {e}. Trying Basic Auth header.") # Method 2: Send credentials as Basic Auth header try: - self._log("Attempting OAuth2 token fetch with Basic Auth header.") + logger.info("Attempting OAuth2 token fetch with Basic Auth header.") header_auth = AiohttpBasicAuth(auth_details.client_id, auth_details.client_secret) header_data = { 'grant_type': 'client_credentials', @@ -311,35 +369,35 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: self._oauth_tokens[client_id] = token_response return token_response["access_token"] except aiohttp.ClientError as e: - self._log(f"OAuth2 with Basic Auth header also failed: {e}", error=True) + logger.error(f"OAuth2 with Basic Auth header also failed: {e}") - def _build_url_with_path_params(self, url_template: str, arguments: Dict[str, Any]) -> str: + def _build_url_with_path_params(self, url_template: str, tool_args: Dict[str, Any]) -> str: """Build URL by substituting path parameters from arguments. Args: url_template: URL template with path parameters in {param_name} format - arguments: Dictionary of arguments that will be modified to remove used path parameters + tool_args: Dictionary of arguments that will be modified to remove used path parameters Returns: URL with path parameters substituted Example: url_template = "https://api.example.com/users/{user_id}/posts/{post_id}" - arguments = {"user_id": "123", "post_id": "456", "limit": "10"} + tool_args = {"user_id": "123", "post_id": "456", "limit": "10"} Returns: "https://api.example.com/users/123/posts/456" - And modifies arguments to: {"limit": "10"} + And modifies tool_args to: {"limit": "10"} """ # Find all path parameters in the URL template path_params = re.findall(r'\{([^}]+)\}', url_template) url = url_template for param_name in path_params: - if param_name in arguments: + if param_name in tool_args: # Replace the parameter in the URL - param_value = str(arguments[param_name]) + param_value = str(tool_args[param_name]) url = url.replace(f'{{{param_name}}}', param_value) # Remove the parameter from arguments so it's not used as a query parameter - arguments.pop(param_name) + tool_args.pop(param_name) else: raise ValueError(f"Missing required path parameter: {param_name}") diff --git a/src/utcp/client/openapi_converter.py b/plugins/communication_protocols/http/src/utcp_http/openapi_converter.py similarity index 68% rename from src/utcp/client/openapi_converter.py rename to plugins/communication_protocols/http/src/utcp_http/openapi_converter.py index 026231a..696600a 100644 --- a/src/utcp/client/openapi_converter.py +++ b/plugins/communication_protocols/http/src/utcp_http/openapi_converter.py @@ -21,25 +21,24 @@ from typing import Any, Dict, List, Optional, Tuple import sys import uuid -from utcp.shared.tool import Tool, ToolInputOutputSchema -from utcp.shared.utcp_manual import UtcpManual from urllib.parse import urlparse - -from utcp.shared.provider import HttpProvider -from utcp.shared.auth import Auth, ApiKeyAuth, BasicAuth, OAuth2Auth - +from utcp.data.auth import Auth +from utcp.data.auth_implementations import ApiKeyAuth, BasicAuth, OAuth2Auth +from utcp.data.utcp_manual import UtcpManual +from utcp.data.tool import Tool, JsonSchema +from utcp_http.http_call_template import HttpCallTemplate class OpenApiConverter: """Converts OpenAPI specifications into UTCP tool definitions. Processes OpenAPI 2.0 and 3.0 specifications to generate equivalent UTCP tools, handling schema resolution, authentication mapping, and proper - HTTP provider configuration. Each operation in the OpenAPI spec becomes + HTTP call_template configuration. Each operation in the OpenAPI spec becomes a UTCP tool with appropriate input/output schemas. Features: - Complete OpenAPI specification parsing - - Recursive JSON reference ($ref) resolution + - Recursive JSON reference ($ref) resolution - Authentication scheme conversion (API key, Basic, OAuth2) - Input parameter and request body handling - Response schema extraction @@ -50,37 +49,37 @@ class OpenApiConverter: Architecture: The converter works by iterating through all paths and operations in the OpenAPI spec, extracting relevant information for each - operation, and creating corresponding UTCP tools with HTTP providers. + operation, and creating corresponding UTCP tools with HTTP call_templates. Attributes: spec: The parsed OpenAPI specification dictionary. spec_url: Optional URL where the specification was retrieved from. placeholder_counter: Counter for generating unique placeholder variables. - provider_name: Normalized name for the provider derived from the spec. + call_template_name: Normalized name for the call_template derived from the spec. """ - def __init__(self, openapi_spec: Dict[str, Any], spec_url: Optional[str] = None, provider_name: Optional[str] = None): + def __init__(self, openapi_spec: Dict[str, Any], spec_url: Optional[str] = None, call_template_name: Optional[str] = None): """Initialize the OpenAPI converter. Args: openapi_spec: Parsed OpenAPI specification as a dictionary. spec_url: Optional URL where the specification was retrieved from. Used for base URL determination if servers are not specified. - provider_name: Optional custom name for the provider. If not + call_template_name: Optional custom name for the call_template. If not provided, derives name from the specification title. """ self.spec = openapi_spec self.spec_url = spec_url # Single counter for all placeholder variables self.placeholder_counter = 0 - # If provider_name is None then get the first word in spec.info.title - if provider_name is None: - title = openapi_spec.get("info", {}).get("title", "openapi_provider_" + uuid.uuid4().hex) + # If call_template_name is None then get the first word in spec.info.title + if call_template_name is None: + title = openapi_spec.get("info", {}).get("title", "openapi_call_template_" + uuid.uuid4().hex) # Replace characters that are invalid for identifiers invalid_chars = " -.,!?'\"\\/()[]{}#@$%^&*+=~`|;:<>" - self.provider_name = ''.join('_' if c in invalid_chars else c for c in title) + self.call_template_name = ''.join('_' if c in invalid_chars else c for c in title) else: - self.provider_name = provider_name + self.call_template_name = call_template_name def _increment_placeholder_counter(self) -> int: """Increments the global counter and returns the new value. @@ -123,39 +122,6 @@ def convert(self) -> UtcpManual: return UtcpManual(tools=tools) - def _resolve_ref(self, ref: str) -> Dict[str, Any]: - """Resolves a local JSON reference.""" - if not ref.startswith('#/'): - raise ValueError(f"External or non-local references are not supported: {ref}") - - parts = ref[2:].split('/') - node = self.spec - for part in parts: - try: - node = node[part] - except (KeyError, TypeError): - raise ValueError(f"Reference not found: {ref}") - return node - - def _resolve_schema(self, schema: Dict[str, Any]) -> Dict[str, Any]: - """Recursively resolves all $refs in a schema object.""" - if isinstance(schema, dict): - if "$ref" in schema: - resolved_ref = self._resolve_ref(schema["$ref"]) - # The resolved reference could itself contain refs, so we recurse - return self._resolve_schema(resolved_ref) - - # Resolve refs in nested properties - new_schema = {} - for key, value in schema.items(): - new_schema[key] = self._resolve_schema(value) - return new_schema - - if isinstance(schema, list): - return [self._resolve_schema(item) for item in schema] - - return schema - def _extract_auth(self, operation: Dict[str, Any]) -> Optional[Auth]: """Extracts authentication information from OpenAPI operation and global security schemes.""" # First check for operation-level security requirements @@ -190,6 +156,37 @@ def _get_security_schemes(self) -> Dict[str, Any]: # OpenAPI 2.0 format return self.spec.get("securityDefinitions", {}) + + def _resolve_ref_path(self, ref: str, visited: Optional[set] = None) -> Dict[str, Any]: + """Resolves a JSON reference path like '#/components/schemas/X' with cycle detection. + + If a cycle is detected, returns a dict that preserves the original + reference ({"$ref": ref}) instead of erasing it. + """ + if not isinstance(ref, str) or not ref.startswith("#/"): + return {} + visited = visited or set() + if ref in visited: + # Break cycles but keep the reference in place + return {"$ref": ref} + visited.add(ref) + parts = ref[2:].split("/") + node: Any = self.spec + try: + for part in parts: + node = node[part] + # Recursively resolve if nested $ref exists + if isinstance(node, dict) and "$ref" in node: + return self._resolve_ref_path(node["$ref"], visited) + return node if isinstance(node, dict) else {} + except Exception: + return {} + + def _resolve_ref_obj(self, obj: Any, visited: Optional[set] = None) -> Any: + """If obj is a $ref dict, resolves it; otherwise returns obj.""" + if isinstance(obj, dict) and "$ref" in obj: + return self._resolve_ref_path(obj["$ref"], visited) + return obj def _create_auth_from_scheme(self, scheme: Dict[str, Any], scheme_name: str) -> Optional[Auth]: """Creates an Auth object from an OpenAPI security scheme.""" @@ -298,18 +295,17 @@ def _create_tool(self, path: str, method: str, operation: Dict[str, Any], base_u description = operation.get("summary") or operation.get("description", "") tags = operation.get("tags", []) - inputs, header_fields, body_field = self._extract_inputs(operation) + inputs, header_fields, body_field = self._extract_inputs(path, operation) outputs = self._extract_outputs(operation) auth = self._extract_auth(operation) - provider_name = self.spec.get("info", {}).get("title", "openapi_provider_" + uuid.uuid4().hex) + call_template_name = self.spec.get("info", {}).get("title", "call_template_" + uuid.uuid4().hex) # Combine base URL and path, ensuring no double slashes full_url = base_url.rstrip('/') + '/' + path.lstrip('/') - provider = HttpProvider( - name=provider_name, - provider_type="http", + call_template = HttpCallTemplate( + name=call_template_name, http_method=method.upper(), url=full_url, body_field=body_field if body_field else None, @@ -323,19 +319,31 @@ def _create_tool(self, path: str, method: str, operation: Dict[str, Any], base_u inputs=inputs, outputs=outputs, tags=tags, - tool_provider=provider + tool_call_template=call_template ) - def _extract_inputs(self, operation: Dict[str, Any]) -> Tuple[ToolInputOutputSchema, List[str], Optional[str]]: - """Extracts input schema, header fields, and body field from an OpenAPI operation.""" - properties = {} + def _extract_inputs(self, path: str, operation: Dict[str, Any]) -> Tuple[JsonSchema, List[str], Optional[str]]: + """Extracts input schema, header fields, and body field from an OpenAPI operation. + + - Merges path-level and operation-level parameters + - Resolves $ref for parameters + - Supports OpenAPI 2.0 body parameters and 3.0 requestBody + """ + properties: Dict[str, Any] = {} required = [] header_fields = [] body_field = None - # Handle parameters (query, header, path, cookie) - for param in operation.get("parameters", []): - param = self._resolve_schema(param) + # Merge path-level and operation-level parameters + path_item = self.spec.get("paths", {}).get(path, {}) if path else {} + all_params = [] + all_params.extend(path_item.get("parameters", []) or []) + all_params.extend(operation.get("parameters", []) or []) + + # Handle parameters (query, header, path, cookie, body) + for param in all_params: + if isinstance(param, dict) and "$ref" in param: + param = self._resolve_ref_path(param["$ref"], set()) or {} param_name = param.get("name") if not param_name: continue @@ -343,11 +351,31 @@ def _extract_inputs(self, operation: Dict[str, Any]) -> Tuple[ToolInputOutputSch if param.get("in") == "header": header_fields.append(param_name) - schema = self._resolve_schema(param.get("schema", {})) + # OpenAPI 2.0 body parameter + if param.get("in") == "body": + body_field = "body" + json_schema = self._resolve_ref_obj(param.get("schema", {}), set()) or {} + properties[body_field] = { + "description": param.get("description", "Request body"), + **json_schema, + } + if param.get("required"): + required.append(body_field) + continue + + # Non-body parameter + schema = self._resolve_ref_obj(param.get("schema", {}), set()) or {} + if not schema: + # OpenAPI 2.0 non-body params use top-level type/items + if "type" in param: + schema["type"] = param.get("type") + if "items" in param: + schema["items"] = param.get("items") + if "enum" in param: + schema["enum"] = param.get("enum") properties[param_name] = { - "type": schema.get("type", "string"), "description": param.get("description", ""), - **schema + **schema, } if param.get("required"): required.append(param_name) @@ -355,51 +383,63 @@ def _extract_inputs(self, operation: Dict[str, Any]) -> Tuple[ToolInputOutputSch # Handle request body request_body = operation.get("requestBody") if request_body: - resolved_body = self._resolve_schema(request_body) - content = resolved_body.get("content", {}) + content = request_body.get("content", {}) json_schema = content.get("application/json", {}).get("schema") + json_schema = self._resolve_ref_obj(json_schema, set()) if json_schema else None if json_schema: # Add a single 'body' field to represent the request body body_field = "body" properties[body_field] = { - "description": resolved_body.get("description", "Request body"), - **self._resolve_schema(json_schema) + "description": json_schema.get("description", "Request body"), + **json_schema } - if resolved_body.get("required"): + if json_schema.get("required"): required.append(body_field) - schema = ToolInputOutputSchema(properties=properties, required=required if required else None) + schema = JsonSchema(properties=properties, required=required if required else None) return schema, header_fields, body_field - def _extract_outputs(self, operation: Dict[str, Any]) -> ToolInputOutputSchema: + def _extract_outputs(self, operation: Dict[str, Any]) -> JsonSchema: """Extracts the output schema from an OpenAPI operation, resolving refs.""" - success_response = operation.get("responses", {}).get("200") or operation.get("responses", {}).get("201") + responses = operation.get("responses", {}) or {} + success_response = responses.get("200") or responses.get("201") or responses.get("default") if not success_response: - return ToolInputOutputSchema() + return JsonSchema() - resolved_response = self._resolve_schema(success_response) - content = resolved_response.get("content", {}) - json_schema = content.get("application/json", {}).get("schema") + json_schema = None + if "content" in success_response: + content = success_response.get("content", {}) + json_schema = content.get("application/json", {}).get("schema") + # Fallback to any content type if application/json missing + if json_schema is None and isinstance(content, dict): + for v in content.values(): + if isinstance(v, dict) and "schema" in v: + json_schema = v.get("schema") + break + elif "schema" in success_response: # OpenAPI 2.0 + json_schema = success_response.get("schema") if not json_schema: - return ToolInputOutputSchema() + return JsonSchema() + + # Resolve $ref in response schema + json_schema = self._resolve_ref_obj(json_schema, set()) or {} - resolved_json_schema = self._resolve_schema(json_schema) schema_args = { - "type": resolved_json_schema.get("type", "object"), - "properties": resolved_json_schema.get("properties", {}), - "required": resolved_json_schema.get("required"), - "description": resolved_json_schema.get("description"), - "title": resolved_json_schema.get("title"), + "type": json_schema.get("type", "object"), + "properties": json_schema.get("properties", {}), + "required": json_schema.get("required"), + "description": json_schema.get("description"), + "title": json_schema.get("title"), } # Handle array item types - if schema_args["type"] == "array" and "items" in resolved_json_schema: - schema_args["items"] = resolved_json_schema.get("items") + if schema_args["type"] == "array" and "items" in json_schema: + schema_args["items"] = json_schema.get("items") # Handle additional schema attributes for attr in ["enum", "minimum", "maximum", "format"]: - if attr in resolved_json_schema: - schema_args[attr] = resolved_json_schema.get(attr) + if attr in json_schema: + schema_args[attr] = json_schema.get(attr) - return ToolInputOutputSchema(**schema_args) + return JsonSchema(**schema_args) diff --git a/plugins/communication_protocols/http/src/utcp_http/sse_call_template.py b/plugins/communication_protocols/http/src/utcp_http/sse_call_template.py new file mode 100644 index 0000000..9414921 --- /dev/null +++ b/plugins/communication_protocols/http/src/utcp_http/sse_call_template.py @@ -0,0 +1,50 @@ +from utcp.data.call_template import CallTemplate, CallTemplateSerializer +from utcp.data.auth import Auth +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback +from typing import Optional, Dict, List, Literal +from pydantic import Field + +class SseCallTemplate(CallTemplate): + """Provider configuration for Server-Sent Events (SSE) tools. + + Enables real-time streaming of events from server to client using the + Server-Sent Events protocol. Supports automatic reconnection and + event type filtering. All tool arguments not mapped to URL body, headers + or query pattern parameters are passed as query parameters using '?arg_name={arg_value}'. + + Attributes: + call_template_type: Always "sse" for SSE providers. + url: The SSE endpoint URL to connect to. + event_type: Optional filter for specific event types. If None, all events are received. + reconnect: Whether to automatically reconnect on connection loss. + retry_timeout: Timeout in milliseconds before attempting reconnection. + auth: Optional authentication configuration. + headers: Optional static headers for the initial connection. + body_field: Optional tool argument name to map to request body during connection. + header_fields: List of tool argument names to map to HTTP headers during connection. + """ + + call_template_type: Literal["sse"] = "sse" + url: str + event_type: Optional[str] = None + reconnect: bool = True + retry_timeout: int = 30000 # Retry timeout in milliseconds if disconnected + auth: Optional[Auth] = None + headers: Optional[Dict[str, str]] = None + body_field: Optional[str] = Field(default=None, description="The name of the single input field to be sent as the request body.") + header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers for the initial connection.") + + +class SSECallTemplateSerializer(Serializer[SseCallTemplate]): + """Serializer for SSECallTemplate.""" + + def to_dict(self, obj: SseCallTemplate) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> SseCallTemplate: + try: + return SseCallTemplate.model_validate(obj) + except Exception as e: + raise UtcpSerializerValidationError("Invalid SSECallTemplate: " + traceback.format_exc()) from e \ No newline at end of file diff --git a/src/utcp/client/transport_interfaces/sse_transport.py b/plugins/communication_protocols/http/src/utcp_http/sse_communication_protocol.py similarity index 66% rename from src/utcp/client/transport_interfaces/sse_transport.py rename to plugins/communication_protocols/http/src/utcp_http/sse_communication_protocol.py index e10bb54..0457efa 100644 --- a/src/utcp/client/transport_interfaces/sse_transport.py +++ b/plugins/communication_protocols/http/src/utcp_http/sse_communication_protocol.py @@ -1,27 +1,36 @@ -from typing import Dict, Any, List, Optional, Callable, AsyncIterator +from typing import Dict, Any, List, Optional, Callable, AsyncIterator, AsyncGenerator import aiohttp import json import asyncio import re import base64 -from utcp.client.client_transport_interface import ClientTransportInterface -from utcp.shared.provider import Provider, SSEProvider -from utcp.shared.tool import Tool -from utcp.shared.utcp_manual import UtcpManual -from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.data.call_template import CallTemplate +from utcp.data.tool import Tool +from utcp.data.utcp_manual import UtcpManual, UtcpManualSerializer +from utcp.data.register_manual_response import RegisterManualResult +from utcp.data.auth_implementations.api_key_auth import ApiKeyAuth +from utcp.data.auth_implementations.basic_auth import BasicAuth +from utcp.data.auth_implementations.oauth2_auth import OAuth2Auth +from utcp_http.sse_call_template import SseCallTemplate from aiohttp import ClientSession, BasicAuth as AiohttpBasicAuth +import traceback +import logging +logger = logging.getLogger(__name__) -class SSEClientTransport(ClientTransportInterface): - """Client transport implementation for Server-Sent Events providers.""" +class SseCommunicationProtocol(CommunicationProtocol): + """SSE communication protocol implementation for UTCP client. + + Handles Server-Sent Events based tool providers with streaming capabilities. + """ def __init__(self, logger: Optional[Callable[[str], None]] = None): self._oauth_tokens: Dict[str, Dict[str, Any]] = {} - self._log = logger or (lambda *args, **kwargs: None) self._active_connections: Dict[str, tuple[aiohttp.ClientResponse, aiohttp.ClientSession]] = {} - def _apply_auth(self, provider: SSEProvider, headers: Dict[str, str], query_params: Dict[str, Any]) -> tuple: + def _apply_auth(self, provider: SseCallTemplate, headers: Dict[str, str], query_params: Dict[str, Any]) -> tuple: """Apply authentication to the request based on the provider's auth configuration. Returns: @@ -40,7 +49,7 @@ def _apply_auth(self, provider: SSEProvider, headers: Dict[str, str], query_para elif provider.auth.location == "cookie": cookies[provider.auth.var_name] = provider.auth.api_key else: - self._log("API key not found for ApiKeyAuth.", error=True) + logger.error("API key not found for ApiKeyAuth.") raise ValueError("API key for ApiKeyAuth not found.") elif isinstance(provider.auth, BasicAuth): @@ -53,13 +62,13 @@ def _apply_auth(self, provider: SSEProvider, headers: Dict[str, str], query_para return auth, cookies - async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: - """Discover tools from an SSE provider.""" - if not isinstance(manual_provider, SSEProvider): - raise ValueError("SSEClientTransport can only be used with SSEProvider") + async def register_manual(self, caller, manual_call_template: CallTemplate) -> RegisterManualResult: + """Register a manual and its tools from an SSE provider.""" + if not isinstance(manual_call_template, SseCallTemplate): + raise ValueError("SSECommunicationProtocol can only be used with SSECallTemplate") try: - url = manual_provider.url + url = manual_call_template.url # Security check: Enforce HTTPS or localhost to prevent MITM attacks if not (url.startswith("https://") or url.startswith("http://localhost") or url.startswith("http://127.0.0.1")): @@ -68,23 +77,23 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: "Non-secure URLs are vulnerable to man-in-the-middle attacks." ) - self._log(f"Discovering tools from '{manual_provider.name}' (SSE) at {url}") + logger.info(f"Discovering tools from '{manual_call_template.name}' (SSE) at {url}") # Use the provider's configuration (headers, auth, etc.) - request_headers = manual_provider.headers.copy() if manual_provider.headers else {} + request_headers = manual_call_template.headers.copy() if manual_call_template.headers else {} body_content = None # Handle authentication query_params: Dict[str, Any] = {} - auth, cookies = self._apply_auth(manual_provider, request_headers, query_params) + auth, cookies = self._apply_auth(manual_call_template, request_headers, query_params) # Handle OAuth2 separately as it's async - if isinstance(manual_provider.auth, OAuth2Auth): - token = await self._handle_oauth2(manual_provider.auth) + if isinstance(manual_call_template.auth, OAuth2Auth): + token = await self._handle_oauth2(manual_call_template.auth) request_headers["Authorization"] = f"Bearer {token}" # Handle body content if specified - if manual_provider.body_field: + if manual_call_template.body_field: # For discovery, we typically don't have body content, but support it if needed body_content = None @@ -118,50 +127,70 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: ) as response: response.raise_for_status() response_data = await response.json() - utcp_manual = UtcpManual(**response_data) - return utcp_manual.tools + utcp_manual = UtcpManualSerializer().validate_dict(response_data) + return RegisterManualResult( + success=True, + manual_call_template=manual_call_template, + manual=utcp_manual, + errors=[] + ) except Exception as e: - self._log(f"Error discovering tools from '{manual_provider.name}': {e}", error=True) - return [] + logger.error(f"Error discovering tools from '{manual_call_template.name}': {e}") + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[traceback.format_exc()] + ) - async def deregister_tool_provider(self, manual_provider: Provider) -> None: - """Deregister an SSE provider and close any active connections.""" - if manual_provider.name in self._active_connections: - self._log(f"Closing active SSE connection for provider '{manual_provider.name}'") - response, session = self._active_connections.pop(manual_provider.name) + async def deregister_manual(self, caller, manual_call_template: CallTemplate) -> None: + """Deregister an SSE manual and close any active connections.""" + template_name = manual_call_template.name + if template_name in self._active_connections: + response, session = self._active_connections.pop(template_name) response.close() await session.close() - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> AsyncIterator[Any]: - """Calls a tool on an SSE provider and returns an async iterator for the events.""" - if not isinstance(tool_provider, SSEProvider): - raise ValueError("SSEClientTransport can only be used with SSEProvider") + async def call_tool(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + """Execute a tool call through SSE transport.""" + if not isinstance(tool_call_template, SseCallTemplate): + raise ValueError("SSECommunicationProtocol can only be used with SSECallTemplate") + + event_list = [] + async for event in self.call_tool_streaming(caller, tool_name, tool_args, tool_call_template): + event_list.append(event) + return event_list + + async def call_tool_streaming(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> AsyncGenerator[Any, None]: + """Execute a tool call through SSE transport with streaming.""" + if not isinstance(tool_call_template, SseCallTemplate): + raise ValueError("SSECommunicationProtocol can only be used with SSECallTemplate") - request_headers = tool_provider.headers.copy() if tool_provider.headers else {} + request_headers = tool_call_template.headers.copy() if tool_call_template.headers else {} body_content = None - remaining_args = arguments.copy() + remaining_args = tool_args.copy() request_headers["Accept"] = "text/event-stream" - if tool_provider.header_fields: - for field_name in tool_provider.header_fields: + if tool_call_template.header_fields: + for field_name in tool_call_template.header_fields: if field_name in remaining_args: request_headers[field_name] = str(remaining_args.pop(field_name)) - if tool_provider.body_field and tool_provider.body_field in remaining_args: - body_content = remaining_args.pop(tool_provider.body_field) + if tool_call_template.body_field and tool_call_template.body_field in remaining_args: + body_content = remaining_args.pop(tool_call_template.body_field) # Build the URL with path parameters substituted - url = self._build_url_with_path_params(tool_provider.url, remaining_args) + url = self._build_url_with_path_params(tool_call_template.url, remaining_args) # The rest of the arguments are query parameters query_params = remaining_args # Handle authentication - auth, cookies = self._apply_auth(tool_provider, request_headers, query_params) + auth, cookies = self._apply_auth(tool_call_template, request_headers, query_params) # Handle OAuth2 separately as it's async - if isinstance(tool_provider.auth, OAuth2Auth): - token = await self._handle_oauth2(tool_provider.auth) + if isinstance(tool_call_template.auth, OAuth2Auth): + token = await self._handle_oauth2(tool_call_template.auth) request_headers["Authorization"] = f"Bearer {token}" session = aiohttp.ClientSession() @@ -175,11 +204,12 @@ async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provid auth=auth, cookies=cookies, json=json_data, data=data, timeout=None ) response.raise_for_status() - self._active_connections[tool_provider.name] = (response, session) - return self._process_sse_stream(response, tool_provider.event_type) + self._active_connections[tool_call_template.name] = (response, session) + async for event in self._process_sse_stream(response, tool_call_template.event_type): + yield event except Exception as e: await session.close() - self._log(f"Error establishing SSE connection to '{tool_provider.name}': {e}", error=True) + logger.error(f"Error establishing SSE connection to '{tool_call_template.name}': {e}") raise async def _process_sse_stream(self, response: aiohttp.ClientResponse, event_type=None): @@ -231,7 +261,7 @@ async def _process_sse_stream(self, response: aiohttp.ClientResponse, event_type except json.JSONDecodeError: yield current_event['data'] except Exception as e: - self._log(f"Error processing SSE stream: {e}", error=True) + logger.error(f"Error processing SSE stream: {e}") raise finally: pass # Session is managed and closed by deregister_tool_provider @@ -251,7 +281,7 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: self._oauth_tokens[client_id] = token_response return token_response["access_token"] except aiohttp.ClientError as e: - self._log(f"OAuth2 with body failed: {e}. Trying Basic Auth.") + logger.error(f"OAuth2 with body failed: {e}. Trying Basic Auth.") try: # Method 2: Credentials in header header_auth = aiohttp.BasicAuth(client_id, auth_details.client_secret) @@ -262,7 +292,7 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: self._oauth_tokens[client_id] = token_response return token_response["access_token"] except aiohttp.ClientError as e: - self._log(f"OAuth2 with header failed: {e}", error=True) + logger.error(f"OAuth2 with header failed: {e}") raise e async def close(self): @@ -274,33 +304,33 @@ async def close(self): await session.close() self._active_connections.clear() - def _build_url_with_path_params(self, url_template: str, arguments: Dict[str, Any]) -> str: + def _build_url_with_path_params(self, url_template: str, tool_args: Dict[str, Any]) -> str: """Build URL by substituting path parameters from arguments. Args: url_template: URL template with path parameters in {param_name} format - arguments: Dictionary of arguments that will be modified to remove used path parameters + tool_args: Dictionary of arguments that will be modified to remove used path parameters Returns: URL with path parameters substituted Example: url_template = "https://api.example.com/users/{user_id}/posts/{post_id}" - arguments = {"user_id": "123", "post_id": "456", "limit": "10"} + tool_args = {"user_id": "123", "post_id": "456", "limit": "10"} Returns: "https://api.example.com/users/123/posts/456" - And modifies arguments to: {"limit": "10"} + And modifies tool_args to: {"limit": "10"} """ # Find all path parameters in the URL template path_params = re.findall(r'\{([^}]+)\}', url_template) url = url_template for param_name in path_params: - if param_name in arguments: + if param_name in tool_args: # Replace the parameter in the URL - param_value = str(arguments[param_name]) + param_value = str(tool_args[param_name]) url = url.replace(f'{{{param_name}}}', param_value) # Remove the parameter from arguments so it's not used as a query parameter - arguments.pop(param_name) + tool_args.pop(param_name) else: raise ValueError(f"Missing required path parameter: {param_name}") diff --git a/plugins/communication_protocols/http/src/utcp_http/streamable_http_call_template.py b/plugins/communication_protocols/http/src/utcp_http/streamable_http_call_template.py new file mode 100644 index 0000000..caf62ae --- /dev/null +++ b/plugins/communication_protocols/http/src/utcp_http/streamable_http_call_template.py @@ -0,0 +1,52 @@ +from utcp.data.call_template import CallTemplate, CallTemplateSerializer +from utcp.data.auth import Auth +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback +from typing import Optional, Dict, List, Literal +from pydantic import Field + +class StreamableHttpCallTemplate(CallTemplate): + """Provider configuration for HTTP streaming tools. + + Uses HTTP Chunked Transfer Encoding to enable streaming of large responses + or real-time data. Useful for tools that return large datasets or provide + progressive results. All tool arguments not mapped to URL body, headers + or query pattern parameters are passed as query parameters using '?arg_name={arg_value}'. + + Attributes: + call_template_type: Always "streamable_http" for HTTP streaming providers. + url: The streaming HTTP endpoint URL. Supports path parameters. + http_method: The HTTP method to use (GET or POST). + content_type: The Content-Type header for requests. + chunk_size: Size of each chunk in bytes for reading the stream. + timeout: Request timeout in milliseconds. + headers: Optional static headers to include in requests. + auth: Optional authentication configuration. + body_field: Optional tool argument name to map to HTTP request body. + header_fields: List of tool argument names to map to HTTP request headers. + """ + + call_template_type: Literal["streamable_http"] = "streamable_http" + url: str + http_method: Literal["GET", "POST"] = "GET" + content_type: str = "application/octet-stream" + chunk_size: int = 4096 # Size of chunks in bytes + timeout: int = 60000 # Timeout in milliseconds + headers: Optional[Dict[str, str]] = None + auth: Optional[Auth] = None + body_field: Optional[str] = Field(default=None, description="The name of the single input field to be sent as the request body.") + header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers.") + + +class StreamableHttpCallTemplateSerializer(Serializer[StreamableHttpCallTemplate]): + """Serializer for StreamableHttpCallTemplate.""" + + def to_dict(self, obj: StreamableHttpCallTemplate) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> StreamableHttpCallTemplate: + try: + return StreamableHttpCallTemplate.model_validate(obj) + except Exception as e: + raise UtcpSerializerValidationError("Invalid StreamableHttpCallTemplate: " + traceback.format_exc()) from e diff --git a/src/utcp/client/transport_interfaces/streamable_http_transport.py b/plugins/communication_protocols/http/src/utcp_http/streamable_http_communication_protocol.py similarity index 56% rename from src/utcp/client/transport_interfaces/streamable_http_transport.py rename to plugins/communication_protocols/http/src/utcp_http/streamable_http_communication_protocol.py index b092d08..d15cd51 100644 --- a/src/utcp/client/transport_interfaces/streamable_http_transport.py +++ b/plugins/communication_protocols/http/src/utcp_http/streamable_http_communication_protocol.py @@ -1,25 +1,33 @@ -from typing import Dict, Any, List, Optional, Callable, AsyncIterator, Tuple +from typing import Dict, Any, List, Optional, Callable, AsyncIterator, Tuple, AsyncGenerator import aiohttp import json import re -from utcp.client.client_transport_interface import ClientTransportInterface -from utcp.shared.provider import Provider, StreamableHttpProvider -from utcp.shared.tool import Tool -from utcp.shared.utcp_manual import UtcpManual -from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.data.call_template import CallTemplate +from utcp.data.tool import Tool +from utcp.data.utcp_manual import UtcpManual, UtcpManualSerializer +from utcp.data.register_manual_response import RegisterManualResult +from utcp.data.auth_implementations import ApiKeyAuth +from utcp.data.auth_implementations import BasicAuth +from utcp.data.auth_implementations import OAuth2Auth +from utcp_http.streamable_http_call_template import StreamableHttpCallTemplate from aiohttp import ClientSession, BasicAuth as AiohttpBasicAuth, ClientResponse +import logging +logger = logging.getLogger(__name__) -class StreamableHttpClientTransport(ClientTransportInterface): - """Client transport implementation for HTTP streaming (chunked transfer encoding) providers using aiohttp.""" +class StreamableHttpCommunicationProtocol(CommunicationProtocol): + """Streamable HTTP communication protocol implementation for UTCP client. + + Handles HTTP streaming with chunked transfer encoding for real-time data. + """ - def __init__(self, logger: Optional[Callable[[str, Any], None]] = None): + def __init__(self): self._oauth_tokens: Dict[str, Dict[str, Any]] = {} - self._log = logger or (lambda *args, **kwargs: None) self._active_connections: Dict[str, Tuple[ClientResponse, ClientSession]] = {} - def _apply_auth(self, provider: StreamableHttpProvider, headers: Dict[str, str], query_params: Dict[str, Any]) -> tuple: + def _apply_auth(self, provider: StreamableHttpCallTemplate, headers: Dict[str, str], query_params: Dict[str, Any]) -> tuple: """Apply authentication to the request based on the provider's auth configuration. Returns: @@ -38,7 +46,7 @@ def _apply_auth(self, provider: StreamableHttpProvider, headers: Dict[str, str], elif provider.auth.location == "cookie": cookies[provider.auth.var_name] = provider.auth.api_key else: - self._log("API key not found for ApiKeyAuth.", error=True) + logger.error("API key not found for ApiKeyAuth.") raise ValueError("API key for ApiKeyAuth not found.") elif isinstance(provider.auth, BasicAuth): @@ -53,9 +61,9 @@ def _apply_auth(self, provider: StreamableHttpProvider, headers: Dict[str, str], async def close(self): """Close all active connections and clear internal state.""" - self._log("Closing all active HTTP stream connections.") + logger.info("Closing all active HTTP stream connections.") for provider_name, (response, session) in list(self._active_connections.items()): - self._log(f"Closing connection for provider: {provider_name}") + logger.info(f"Closing connection for provider: {provider_name}") if not response.closed: response.close() # Close the response if not session.closed: @@ -63,12 +71,12 @@ async def close(self): self._active_connections.clear() self._oauth_tokens.clear() - async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: - """Discover tools from a StreamableHttp provider.""" - if not isinstance(manual_provider, StreamableHttpProvider): - raise ValueError("StreamableHttpClientTransport can only be used with StreamableHttpProvider") + async def register_manual(self, caller, manual_call_template: CallTemplate) -> RegisterManualResult: + """Register a manual and its tools from a StreamableHttp provider.""" + if not isinstance(manual_call_template, StreamableHttpCallTemplate): + raise ValueError("StreamableHttpCommunicationProtocol can only be used with StreamableHttpCallTemplate") - url = manual_provider.url + url = manual_call_template.url # Security check: Enforce HTTPS or localhost to prevent MITM attacks if not (url.startswith("https://") or url.startswith("http://localhost") or url.startswith("http://127.0.0.1")): @@ -77,31 +85,31 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: "Non-secure URLs are vulnerable to man-in-the-middle attacks." ) - self._log(f"Discovering tools from '{manual_provider.name}' (HTTP Stream) at {url}") + logger.info(f"Discovering tools from '{manual_call_template.name}' (HTTP Stream) at {url}") try: - # Use the provider's configuration (headers, auth, etc.) - request_headers = manual_provider.headers.copy() if manual_provider.headers else {} + # Use the template's configuration (headers, auth, etc.) + request_headers = manual_call_template.headers.copy() if manual_call_template.headers else {} body_content = None # Handle authentication query_params: Dict[str, Any] = {} - auth, cookies = self._apply_auth(manual_provider, request_headers, query_params) + auth, cookies = self._apply_auth(manual_call_template, request_headers, query_params) # Handle OAuth2 separately as it's async - if isinstance(manual_provider.auth, OAuth2Auth): - token = await self._handle_oauth2(manual_provider.auth) + if isinstance(manual_call_template.auth, OAuth2Auth): + token = await self._handle_oauth2(manual_call_template.auth) request_headers["Authorization"] = f"Bearer {token}" # Handle body content if specified - if manual_provider.body_field: + if manual_call_template.body_field: # For discovery, we typically don't have body content, but support it if needed body_content = None async with aiohttp.ClientSession() as session: # Set content-type header if body is provided and header not already set if body_content is not None and "Content-Type" not in request_headers: - request_headers["Content-Type"] = manual_provider.content_type + request_headers["Content-Type"] = manual_call_template.content_type # Prepare body content based on content type data = None @@ -112,8 +120,8 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: else: data = body_content - # Make the request with the provider's HTTP method - method = manual_provider.http_method.lower() + # Make the request with the template's HTTP method + method = manual_call_template.http_method.lower() request_method = getattr(session, method) async with request_method( @@ -128,79 +136,120 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: ) as response: response.raise_for_status() response_data = await response.json() - utcp_manual = UtcpManual(**response_data) - return utcp_manual.tools + utcp_manual = UtcpManualSerializer().validate_dict(response_data) + return RegisterManualResult( + success=True, + manual_call_template=manual_call_template, + manual=utcp_manual, + errors=[] + ) except aiohttp.ClientResponseError as e: - self._log(f"Error discovering tools from '{manual_provider.name}': {e.status}, message='{e.message}', url='{e.request_info.url}'", error=True) - return [] + error_msg = f"Error discovering tools from '{manual_call_template.name}': {e.status}, message='{e.message}', url='{e.request_info.url}'" + logger.error(error_msg) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[error_msg] + ) except (json.JSONDecodeError, aiohttp.ClientError) as e: - self._log(f"Error processing request for '{manual_provider.name}': {e}", error=True) - return [] + error_msg = f"Error processing request for '{manual_call_template.name}': {e}" + logger.error(error_msg) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[error_msg] + ) except Exception as e: - self._log(f"An unexpected error occurred while discovering tools from '{manual_provider.name}': {e}", error=True) - return [] - - async def deregister_tool_provider(self, manual_provider: Provider) -> None: - """Deregister a StreamableHttp provider and close any active connections.""" - if not isinstance(manual_provider, StreamableHttpProvider): - return + error_msg = f"An unexpected error occurred while discovering tools from '{manual_call_template.name}': {e}" + logger.error(error_msg) + return RegisterManualResult( + success=False, + manual_call_template=manual_call_template, + manual=UtcpManual(utcp_version="1.0.0", manual_version="0.0.0", tools=[]), + errors=[error_msg] + ) - if manual_provider.name in self._active_connections: - self._log(f"Closing active HTTP stream connection for provider '{manual_provider.name}'") - response, session = self._active_connections.pop(manual_provider.name) + async def deregister_manual(self, caller, manual_call_template: CallTemplate) -> None: + """Deregister a StreamableHttp manual and close any active connections.""" + template_name = manual_call_template.name + if template_name in self._active_connections: + logger.info(f"Closing active HTTP stream connection for template '{template_name}'") + response, session = self._active_connections.pop(template_name) if not response.closed: response.close() if not session.closed: await session.close() + else: + logger.info(f"No active connection found for template '{template_name}'") - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> AsyncIterator[Any]: - """Calls a tool on a StreamableHttp provider and returns an async iterator for the response chunks.""" - if not isinstance(tool_provider, StreamableHttpProvider): - raise ValueError("StreamableHttpClientTransport can only be used with StreamableHttpProvider") + async def call_tool(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + """Execute a tool call through StreamableHttp transport.""" + if not isinstance(tool_call_template, StreamableHttpCallTemplate): + raise ValueError("StreamableHttpCommunicationProtocol can only be used with StreamableHttpCallTemplate") + + is_bytes = False + chunk_list = [] + chunk_bytes = b'' + async for chunk in self.call_tool_streaming(caller, tool_name, tool_args, tool_call_template): + if isinstance(chunk, bytes): + is_bytes = True + chunk_bytes += chunk + else: + chunk_list.append(chunk) + if is_bytes: + return chunk_bytes + return chunk_list + + async def call_tool_streaming(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> AsyncGenerator[Any, None]: + """Execute a tool call through StreamableHttp transport with streaming.""" + if not isinstance(tool_call_template, StreamableHttpCallTemplate): + raise ValueError("StreamableHttpCommunicationProtocol can only be used with StreamableHttpCallTemplate") - request_headers = tool_provider.headers.copy() if tool_provider.headers else {} + request_headers = tool_call_template.headers.copy() if tool_call_template.headers else {} body_content = None - remaining_args = arguments.copy() + remaining_args = tool_args.copy() - if tool_provider.header_fields: - for field_name in tool_provider.header_fields: + if tool_call_template.header_fields: + for field_name in tool_call_template.header_fields: if field_name in remaining_args: request_headers[field_name] = str(remaining_args.pop(field_name)) - if tool_provider.body_field and tool_provider.body_field in remaining_args: - body_content = remaining_args.pop(tool_provider.body_field) + if tool_call_template.body_field and tool_call_template.body_field in remaining_args: + body_content = remaining_args.pop(tool_call_template.body_field) # Build the URL with path parameters substituted - url = self._build_url_with_path_params(tool_provider.url, remaining_args) + url = self._build_url_with_path_params(tool_call_template.url, remaining_args) # The rest of the arguments are query parameters query_params = remaining_args # Handle authentication - auth_handler, cookies = self._apply_auth(tool_provider, request_headers, query_params) + auth_handler, cookies = self._apply_auth(tool_call_template, request_headers, query_params) # Handle OAuth2 separately as it's async - if isinstance(tool_provider.auth, OAuth2Auth): - token = await self._handle_oauth2(tool_provider.auth) + if isinstance(tool_call_template.auth, OAuth2Auth): + token = await self._handle_oauth2(tool_call_template.auth) request_headers["Authorization"] = f"Bearer {token}" session = ClientSession() try: - timeout_seconds = tool_provider.timeout / 1000 if tool_provider.timeout else 60.0 + timeout_seconds = tool_call_template.timeout / 1000 if tool_call_template.timeout else 60.0 timeout = aiohttp.ClientTimeout(total=timeout_seconds) data = None json_data = None if body_content is not None: if "Content-Type" not in request_headers: - request_headers["Content-Type"] = tool_provider.content_type + request_headers["Content-Type"] = tool_call_template.content_type if "application/json" in request_headers.get("Content-Type", ""): json_data = body_content else: data = body_content response = await session.request( - method=tool_provider.http_method, + method=tool_call_template.http_method, url=url, params=query_params, headers=request_headers, @@ -212,12 +261,13 @@ async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provid ) response.raise_for_status() - self._active_connections[tool_provider.name] = (response, session) - return self._process_http_stream(response, tool_provider.chunk_size, tool_provider.name) + self._active_connections[tool_call_template.name] = (response, session) + async for chunk in self._process_http_stream(response, tool_call_template.chunk_size, tool_call_template.name): + yield chunk except Exception as e: await session.close() - self._log(f"Error establishing HTTP stream connection to '{tool_provider.name}': {e}", error=True) + logger.error(f"Error establishing HTTP stream connection to '{tool_call_template.name}': {e}") raise async def _process_http_stream(self, response: ClientResponse, chunk_size: Optional[int], provider_name: str) -> AsyncIterator[Any]: @@ -231,7 +281,7 @@ async def _process_http_stream(self, response: ClientResponse, chunk_size: Optio try: yield json.loads(line) except json.JSONDecodeError: - self._log(f"Error parsing NDJSON line for '{provider_name}': {line[:100]}", error=True) + logger.error(f"Error parsing NDJSON line for '{provider_name}': {line[:100]}") yield line # Yield raw line on error elif 'application/octet-stream' in content_type: async for chunk in response.content.iter_chunked(chunk_size or 8192): @@ -246,7 +296,7 @@ async def _process_http_stream(self, response: ClientResponse, chunk_size: Optio try: yield json.loads(buffer) except json.JSONDecodeError: - self._log(f"Error parsing JSON response for '{provider_name}': {buffer[:100]}", error=True) + logger.error(f"Error parsing JSON response for '{provider_name}': {buffer[:100]}") yield buffer # Yield raw buffer on error else: # Default to binary chunk streaming for unknown content types @@ -254,7 +304,7 @@ async def _process_http_stream(self, response: ClientResponse, chunk_size: Optio if chunk: yield chunk except Exception as e: - self._log(f"Error processing HTTP stream for '{provider_name}': {e}", error=True) + logger.error(f"Error processing HTTP stream for '{provider_name}': {e}") raise finally: # The session is closed later by deregister_tool_provider or close() @@ -272,18 +322,18 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: async with aiohttp.ClientSession() as session: # Method 1: Credentials in body try: - self._log(f"Attempting OAuth2 token fetch for '{client_id}' with credentials in body.") + logger.info(f"Attempting OAuth2 token fetch for '{client_id}' with credentials in body.") async with session.post(auth_details.token_url, data={'grant_type': 'client_credentials', 'client_id': client_id, 'client_secret': auth_details.client_secret, 'scope': auth_details.scope}) as response: response.raise_for_status() token_data = await response.json() self._oauth_tokens[client_id] = token_data return token_data['access_token'] except aiohttp.ClientError as e: - self._log(f"OAuth2 with credentials in body failed: {e}. Trying Basic Auth header.") + logger.error(f"OAuth2 with credentials in body failed: {e}. Trying Basic Auth header.") # Method 2: Credentials as Basic Auth header try: - self._log(f"Attempting OAuth2 token fetch for '{client_id}' with Basic Auth header.") + logger.info(f"Attempting OAuth2 token fetch for '{client_id}' with Basic Auth header.") auth = AiohttpBasicAuth(client_id, auth_details.client_secret) async with session.post(auth_details.token_url, data={'grant_type': 'client_credentials', 'scope': auth_details.scope}, auth=auth) as response: response.raise_for_status() @@ -291,36 +341,36 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: self._oauth_tokens[client_id] = token_data return token_data['access_token'] except aiohttp.ClientError as e: - self._log(f"OAuth2 with Basic Auth header also failed: {e}", error=True) + logger.error(f"OAuth2 with Basic Auth header also failed: {e}") raise e - def _build_url_with_path_params(self, url_template: str, arguments: Dict[str, Any]) -> str: + def _build_url_with_path_params(self, url_template: str, tool_args: Dict[str, Any]) -> str: """Build URL by substituting path parameters from arguments. Args: url_template: URL template with path parameters in {param_name} format - arguments: Dictionary of arguments that will be modified to remove used path parameters + tool_args: Dictionary of arguments that will be modified to remove used path parameters Returns: URL with path parameters substituted Example: url_template = "https://api.example.com/users/{user_id}/posts/{post_id}" - arguments = {"user_id": "123", "post_id": "456", "limit": "10"} + tool_args = {"user_id": "123", "post_id": "456", "limit": "10"} Returns: "https://api.example.com/users/123/posts/456" - And modifies arguments to: {"limit": "10"} + And modifies tool_args to: {"limit": "10"} """ # Find all path parameters in the URL template path_params = re.findall(r'\{([^}]+)\}', url_template) url = url_template for param_name in path_params: - if param_name in arguments: + if param_name in tool_args: # Replace the parameter in the URL - param_value = str(arguments[param_name]) + param_value = str(tool_args[param_name]) url = url.replace(f'{{{param_name}}}', param_value) # Remove the parameter from arguments so it's not used as a query parameter - arguments.pop(param_name) + tool_args.pop(param_name) else: raise ValueError(f"Missing required path parameter: {param_name}") diff --git a/tests/client/transport_interfaces/sample_tools.json b/plugins/communication_protocols/http/tests/sample_tools.json similarity index 100% rename from tests/client/transport_interfaces/sample_tools.json rename to plugins/communication_protocols/http/tests/sample_tools.json diff --git a/tests/client/transport_interfaces/test_http_transport.py b/plugins/communication_protocols/http/tests/test_http_communication_protocol.py similarity index 50% rename from tests/client/transport_interfaces/test_http_transport.py rename to plugins/communication_protocols/http/tests/test_http_communication_protocol.py index 38b98a6..753ec8e 100644 --- a/tests/client/transport_interfaces/test_http_transport.py +++ b/plugins/communication_protocols/http/tests/test_http_communication_protocol.py @@ -1,14 +1,14 @@ import pytest import pytest_asyncio -import json import aiohttp from aiohttp import web -from unittest.mock import MagicMock - -from utcp.client.transport_interfaces.http_transport import HttpClientTransport -from utcp.shared.provider import HttpProvider -from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth - +from utcp_http.http_communication_protocol import HttpCommunicationProtocol +from utcp_http.http_call_template import HttpCallTemplate +from utcp.data.auth_implementations import ApiKeyAuth +from utcp.data.auth_implementations import BasicAuth +from utcp.data.auth_implementations import OAuth2Auth +from utcp.data.register_manual_response import RegisterManualResult +from utcp.data.call_template import CallTemplate # Setup test HTTP server @pytest_asyncio.fixture @@ -18,17 +18,17 @@ async def app(): # Setup routes for our test server async def tools_handler(request): - # The execution provider points to the /tool endpoint - execution_provider = { - "provider_type": "http", - "name": "test-http-provider-executor", + # The execution call template points to the /tool endpoint + execution_call_template = { + "call_template_type": "http", + "name": "test-http-call-template-executor", "url": str(request.url.origin()) + "/tool", - "http_method": "GET", - "content_type": "application/json" + "http_method": "GET" } - # Return sample tools JSON - return web.json_response({ - "version": "1.0", + # Return sample UTCP manual JSON + utcp_manual = { + "utcp_version": "1.0.0", + "manual_version": "1.0.0", "tools": [ { "name": "test_tool", @@ -46,10 +46,11 @@ async def tools_handler(request): } }, "tags": [], - "tool_provider": execution_provider + "tool_call_template": execution_call_template } ] - }) + } + return web.json_response(utcp_manual) async def token_handler(request): # OAuth2 token endpoint (credentials in body) @@ -134,6 +135,7 @@ async def error_handler(request): app.router.add_get('/tools', tools_handler) app.router.add_get('/tool', tool_handler) app.router.add_post('/tool', tool_handler) + app.router.add_get('/tool/{param1}', tool_handler) # Add path param route app.router.add_post('/token', token_handler) app.router.add_post('/token_header_auth', token_header_auth_handler) app.router.add_get('/error', error_handler) @@ -146,129 +148,132 @@ async def aiohttp_client(aiohttp_client, app): return await aiohttp_client(app) -@pytest.fixture -def logger(): - """Create a mock logger.""" - return MagicMock() - - -@pytest.fixture -def http_transport(logger): - """Create an HTTP transport instance.""" - return HttpClientTransport(logger=logger) +@pytest_asyncio.fixture +async def http_transport(): + """Create an HTTP communication protocol instance.""" + return HttpCommunicationProtocol() @pytest_asyncio.fixture -async def http_provider(aiohttp_client): - """Create a basic HTTP provider for testing.""" - return HttpProvider( - name="test-http-provider", - url=f"{aiohttp_client.make_url('/tools')}", - http_method="GET", - content_type="application/json" +async def http_call_template(aiohttp_client): + """Create a basic HTTP call template for testing.""" + return HttpCallTemplate( + name="test_call_template", + url=f"http://localhost:{aiohttp_client.port}/tools", + http_method="GET" ) @pytest_asyncio.fixture -async def api_key_provider(aiohttp_client): - """Create an HTTP provider with API key auth.""" - return HttpProvider( - name="api-key-provider", - url=f"{aiohttp_client.make_url('/tool')}", +async def api_key_call_template(aiohttp_client): + """Create an HTTP call template with API key auth.""" + return HttpCallTemplate( + name="api-key-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", http_method="GET", - content_type="application/json", - auth=ApiKeyAuth(var_name="X-API-Key", api_key="test-api-key") + auth=ApiKeyAuth(api_key="test-api-key", var_name="X-API-Key", location="header") ) @pytest_asyncio.fixture -async def basic_auth_provider(aiohttp_client): - """Create an HTTP provider with Basic auth.""" - return HttpProvider( - name="basic-auth-provider", - url=f"{aiohttp_client.make_url('/tool')}", +async def basic_auth_call_template(aiohttp_client): + """Create an HTTP call template with Basic auth.""" + return HttpCallTemplate( + name="basic-auth-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", http_method="GET", - content_type="application/json", auth=BasicAuth(username="user", password="pass") ) @pytest_asyncio.fixture -async def oauth2_provider(aiohttp_client): - """Create an HTTP provider with OAuth2 auth.""" - return HttpProvider( - name="oauth2-provider", - url=f"{aiohttp_client.make_url('/tool')}", +async def oauth2_call_template(aiohttp_client): + """Create an HTTP call template with OAuth2 auth.""" + return HttpCallTemplate( + name="oauth2-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", http_method="GET", - content_type="application/json", auth=OAuth2Auth( client_id="client-id", client_secret="client-secret", - token_url=f"{aiohttp_client.make_url('/token')}", + token_url=f"http://localhost:{aiohttp_client.port}/token", scope="read write" ) ) -# Test register_tool_provider +# Test register_manual @pytest.mark.asyncio -async def test_register_tool_provider(http_transport, http_provider, logger): - """Test registering a tool provider.""" - # Call register_tool_provider - tools = await http_transport.register_tool_provider(http_provider) - - # Verify the result is a list of tools - assert isinstance(tools, list) - assert len(tools) > 0 +async def test_register_manual(http_transport: HttpCommunicationProtocol, http_call_template: HttpCallTemplate): + """Test registering a manual.""" + # Call register_manual + result = await http_transport.register_manual(None, http_call_template) + + # Debug: Print the result details if it failed + if not result.success: + # Make a direct request to see what the server returns + async with aiohttp.ClientSession() as session: + async with session.get(http_call_template.url) as response: + content = await response.text() + print(f"Server response: {content}") + + # Verify the result is a RegisterManualResult + assert isinstance(result, RegisterManualResult) + assert result.manual is not None + assert len(result.manual.tools) > 0, f"Expected tools but got empty list. Success: {result.success}" + assert result.success is True + assert not result.errors # Verify each tool has required fields - tool = tools[0] + tool = result.manual.tools[0] assert tool.name == "test_tool" assert tool.description == "Test tool" assert hasattr(tool, "inputs") assert hasattr(tool, "outputs") -# Test error handling when registering a tool provider +# Test error handling when registering a manual @pytest.mark.asyncio -async def test_register_tool_provider_http_error(http_transport, logger, aiohttp_client): - """Test error handling when registering a tool provider.""" - # Create a provider that points to our error endpoint - error_provider = HttpProvider( - name="error-provider", - url=f"{aiohttp_client.make_url('/error')}", - http_method="GET", - content_type="application/json" +async def test_register_manual_http_error(http_transport, aiohttp_client): + """Test error handling when registering a manual.""" + # Create a call template that points to our error endpoint + error_call_template = HttpCallTemplate( + name="error-call-template", + url=f"http://localhost:{aiohttp_client.port}/error", + http_method="GET" ) # Test the register method with error - tools = await http_transport.register_tool_provider(error_provider) + result = await http_transport.register_manual(None, error_call_template) # Verify the results - assert tools == [] - # Logger should be called with error - logger.assert_called() - -# Test deregister_tool_provider + assert isinstance(result, RegisterManualResult) + assert result.success is False + # On error, we should have a manual but no tools + assert result.manual is not None + assert len(result.manual.tools) == 0 + assert result.errors + assert isinstance(result.errors[0], str) + +# Test deregister_manual @pytest.mark.asyncio -async def test_deregister_tool_provider(http_transport, http_provider): - """Test deregistering a tool provider (should be a no-op).""" +async def test_deregister_manual(http_transport, http_call_template): + """Test deregistering a manual (should be a no-op).""" # Deregister should be a no-op - await http_transport.deregister_tool_provider(http_provider) + await http_transport.deregister_manual(None, http_call_template) # Test call_tool_basic @pytest.mark.asyncio -async def test_call_tool_basic(http_transport, http_provider, aiohttp_client): +async def test_call_tool_basic(http_transport, http_call_template, aiohttp_client): """Test calling a tool with basic configuration.""" - # Update provider URL to point to our /tool endpoint - tool_provider = HttpProvider( - name=http_provider.name, - url=f"{aiohttp_client.make_url('/tool')}", - http_method="GET", - content_type=http_provider.content_type + # Update call template URL to point to our /tool endpoint + tool_call_template = HttpCallTemplate( + name=http_call_template.name, + url=f"http://localhost:{aiohttp_client.port}/tool", + http_method="GET" ) # Test calling a tool - result = await http_transport.call_tool("test_tool", {"param1": "value1"}, tool_provider) + result = await http_transport.call_tool(None, "test_tool", {"param1": "value1"}, tool_call_template) # Verify the results assert result == {"result": "success"} @@ -276,10 +281,10 @@ async def test_call_tool_basic(http_transport, http_provider, aiohttp_client): # Test call_tool_with_api_key @pytest.mark.asyncio -async def test_call_tool_with_api_key(http_transport, api_key_provider): +async def test_call_tool_with_api_key(http_transport, api_key_call_template): """Test calling a tool with API key authentication.""" # Test calling a tool with API key auth - result = await http_transport.call_tool("test_tool", {"param1": "value1"}, api_key_provider) + result = await http_transport.call_tool(None, "test_tool", {"param1": "value1"}, api_key_call_template) # Verify result assert result == {"result": "success"} @@ -289,10 +294,10 @@ async def test_call_tool_with_api_key(http_transport, api_key_provider): # Test call_tool_with_basic_auth @pytest.mark.asyncio -async def test_call_tool_with_basic_auth(http_transport, basic_auth_provider): +async def test_call_tool_with_basic_auth(http_transport, basic_auth_call_template): """Test calling a tool with Basic authentication.""" # Test calling a tool with Basic auth - result = await http_transport.call_tool("test_tool", {"param1": "value1"}, basic_auth_provider) + result = await http_transport.call_tool(None, "test_tool", {"param1": "value1"}, basic_auth_call_template) # Verify result assert result == {"result": "success"} @@ -300,10 +305,10 @@ async def test_call_tool_with_basic_auth(http_transport, basic_auth_provider): # Test call_tool_with_oauth2 @pytest.mark.asyncio -async def test_call_tool_with_oauth2(http_transport, oauth2_provider): +async def test_call_tool_with_oauth2(http_transport, oauth2_call_template): """Test calling a tool with OAuth2 authentication (credentials in body).""" # This test uses the primary method (credentials in body) - result = await http_transport.call_tool("test_tool", {"param1": "value1"}, oauth2_provider) + result = await http_transport.call_tool(None, "test_tool", {"param1": "value1"}, oauth2_call_template) assert result == {"result": "success"} @@ -311,16 +316,15 @@ async def test_call_tool_with_oauth2(http_transport, oauth2_provider): @pytest.mark.asyncio async def test_call_tool_with_oauth2_header_auth(http_transport, aiohttp_client): """Test calling a tool with OAuth2 authentication (credentials in header).""" - # This provider points to an endpoint that expects Basic Auth for the token - oauth2_header_provider = HttpProvider( - name="oauth2-header-provider", - url=f"{aiohttp_client.make_url('/tool')}", + # This call template points to an endpoint that expects Basic Auth for the token + oauth2_header_call_template = HttpCallTemplate( + name="oauth2-header-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", http_method="GET", - content_type="application/json", auth=OAuth2Auth( client_id="client-id", client_secret="client-secret", - token_url=f"{aiohttp_client.make_url('/token_header_auth')}", + token_url=f"http://localhost:{aiohttp_client.port}/token_header_auth", scope="read write" ) ) @@ -328,7 +332,7 @@ async def test_call_tool_with_oauth2_header_auth(http_transport, aiohttp_client) # This test uses the fallback method (credentials in header) # The transport will first try the body method, which will fail against this endpoint, # and then it should fall back to the header method and succeed. - result = await http_transport.call_tool("test_tool", {"param1": "value1"}, oauth2_header_provider) + result = await http_transport.call_tool(None, "test_tool", {"param1": "value1"}, oauth2_header_call_template) assert result == {"result": "success"} @@ -337,44 +341,67 @@ async def test_call_tool_with_oauth2_header_auth(http_transport, aiohttp_client) @pytest.mark.asyncio async def test_call_tool_with_body_field(http_transport, aiohttp_client): """Test calling a tool with a body field.""" - # Create provider with body field - provider = HttpProvider( - name="body-field-provider", - url=f"{aiohttp_client.make_url('/tool')}", + # Create call template with body field + call_template = HttpCallTemplate( + name="body-field-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", http_method="POST", - content_type="application/json", body_field="data" ) # Test calling a tool with a body field result = await http_transport.call_tool( + None, "test_tool", {"param1": "value1", "data": {"key": "value"}}, - provider + call_template + ) + + # Verify result + assert result == {"result": "success"} + + +# Test call_tool_with_path_params +@pytest.mark.asyncio +async def test_call_tool_with_path_params(http_transport, aiohttp_client): + """Test calling a tool with path parameters.""" + # Create call template with path params in URL + call_template = HttpCallTemplate( + name="path-params-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool/{{param1}}", + http_method="GET" + ) + + # Test calling a tool with path params + result = await http_transport.call_tool( + None, + "test_tool", + {"param1": "test-value", "param2": "other-value"}, + call_template ) # Verify result assert result == {"result": "success"} -# Test call_tool_with_header_fields +# Test call_tool_with_custom_headers @pytest.mark.asyncio -async def test_call_tool_with_header_fields(http_transport, aiohttp_client): - """Test calling a tool with header fields.""" - # Create provider with header fields - provider = HttpProvider( - name="header-fields-provider", - url=f"{aiohttp_client.make_url('/tool')}", +async def test_call_tool_with_custom_headers(http_transport, aiohttp_client): + """Test calling a tool with custom headers.""" + # Create call template with custom headers + call_template = HttpCallTemplate( + name="custom-headers-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", http_method="GET", - content_type="application/json", - header_fields=["X-Custom-Header"] + additional_headers={"X-Custom-Header": "custom-value"} ) - # Test calling a tool with a header field + # Test calling a tool with custom headers result = await http_transport.call_tool( + None, "test_tool", - {"param1": "value1", "X-Custom-Header": "custom-value"}, - provider + {"param1": "value1"}, + call_template ) # Verify result @@ -383,22 +410,20 @@ async def test_call_tool_with_header_fields(http_transport, aiohttp_client): # Test call_tool_error @pytest.mark.asyncio -async def test_call_tool_error(http_transport, logger, aiohttp_client): +async def test_call_tool_error(http_transport, aiohttp_client): """Test error handling when calling a tool.""" - # Create a provider that will return a DNS error (since the host doesn't exist) - provider = HttpProvider( - name="test-provider", + # Create a call template that will return a DNS error (since the host doesn't exist) + call_template = HttpCallTemplate( + name="test-call-template", url="http://nonexistent.localhost:8080/404", - http_method="GET", - content_type="application/json" + http_method="GET" ) # Test calling a tool that returns a DNS error with pytest.raises(Exception): - await http_transport.call_tool("test_tool", {"param1": "value1"}, provider) + await http_transport.call_tool(None, "test_tool", {"param1": "value1"}, call_template) - # Check that the error was logged - assert logger.call_count >= 1 + # The error should be raised as an exception # Test URL path parameters functionality @@ -473,18 +498,19 @@ async def path_param_handler(request): try: base_url = f"http://localhost:{client.port}" - # Create a provider with path parameters in the URL - provider = HttpProvider( - name="test_provider", + # Create a call template with path parameters in the URL + call_template = HttpCallTemplate( + name="test_call_template", url=f"{base_url}/users/{{user_id}}/posts/{{post_id}}", http_method="GET" ) # Call the tool with path parameters result = await http_transport.call_tool( + None, "get_user_post", {"user_id": "123", "post_id": "456", "limit": "20"}, - provider + call_template ) # Verify the result @@ -492,18 +518,146 @@ async def path_param_handler(request): assert result["post_id"] == "456" assert result["limit"] == "20" assert "Retrieved post 456 for user 123 with limit 20" in result["message"] + finally: # Clean up the test client await client.close() +# Streaming tests: call_tool_streaming should yield a single element equal to call_tool result + + +@pytest.mark.asyncio +async def test_call_tool_streaming_basic(http_transport, http_call_template, aiohttp_client): + """Streaming basic call should yield one result identical to call_tool.""" + tool_call_template = HttpCallTemplate( + name=http_call_template.name, + url=f"http://localhost:{aiohttp_client.port}/tool", + http_method="GET", + ) + stream = http_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, tool_call_template) + results = [chunk async for chunk in stream] + assert results == [{"result": "success"}] + + +@pytest.mark.asyncio +async def test_call_tool_streaming_with_api_key(http_transport, api_key_call_template): + """Streaming with API key auth yields one aggregated result.""" + stream = http_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, api_key_call_template) + results = [chunk async for chunk in stream] + assert results == [{"result": "success"}] + + +@pytest.mark.asyncio +async def test_call_tool_streaming_with_basic_auth(http_transport, basic_auth_call_template): + """Streaming with Basic auth yields one aggregated result.""" + stream = http_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, basic_auth_call_template) + results = [chunk async for chunk in stream] + assert results == [{"result": "success"}] + + +@pytest.mark.asyncio +async def test_call_tool_streaming_with_oauth2(http_transport, oauth2_call_template): + """Streaming with OAuth2 (credentials in body) yields one aggregated result.""" + stream = http_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, oauth2_call_template) + results = [chunk async for chunk in stream] + assert results == [{"result": "success"}] + + +@pytest.mark.asyncio +async def test_call_tool_streaming_with_oauth2_header_auth(http_transport, aiohttp_client): + """Streaming with OAuth2 (credentials in header) yields one aggregated result.""" + oauth2_header_call_template = HttpCallTemplate( + name="oauth2-header-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", + http_method="GET", + auth=OAuth2Auth( + client_id="client-id", + client_secret="client-secret", + token_url=f"http://localhost:{aiohttp_client.port}/token_header_auth", + scope="read write", + ), + ) + stream = http_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, oauth2_header_call_template) + results = [chunk async for chunk in stream] + assert results == [{"result": "success"}] + + +@pytest.mark.asyncio +async def test_call_tool_streaming_with_body_field(http_transport, aiohttp_client): + """Streaming POST with body_field yields one aggregated result.""" + call_template = HttpCallTemplate( + name="body-field-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", + http_method="POST", + body_field="data", + ) + stream = http_transport.call_tool_streaming( + None, + "test_tool", + {"param1": "value1", "data": {"key": "value"}}, + call_template, + ) + results = [chunk async for chunk in stream] + assert results == [{"result": "success"}] + + +@pytest.mark.asyncio +async def test_call_tool_streaming_with_path_params(http_transport, aiohttp_client): + """Streaming with URL path params yields one aggregated result.""" + call_template = HttpCallTemplate( + name="path-params-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool/{{param1}}", + http_method="GET", + ) + stream = http_transport.call_tool_streaming( + None, + "test_tool", + {"param1": "test-value", "param2": "other-value"}, + call_template, + ) + results = [chunk async for chunk in stream] + assert results == [{"result": "success"}] + + +@pytest.mark.asyncio +async def test_call_tool_streaming_with_custom_headers(http_transport, aiohttp_client): + """Streaming with additional headers yields one aggregated result.""" + call_template = HttpCallTemplate( + name="custom-headers-call-template", + url=f"http://localhost:{aiohttp_client.port}/tool", + http_method="GET", + additional_headers={"X-Custom-Header": "custom-value"}, + ) + stream = http_transport.call_tool_streaming( + None, + "test_tool", + {"param1": "value1"}, + call_template, + ) + results = [chunk async for chunk in stream] + assert results == [{"result": "success"}] + + +@pytest.mark.asyncio +async def test_call_tool_streaming_error(http_transport): + """Streaming should propagate errors from call_tool (no elements yielded).""" + call_template = HttpCallTemplate( + name="test-call-template", + url="http://nonexistent.localhost:8080/404", + http_method="GET", + ) + with pytest.raises(Exception): + async for _ in http_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, call_template): + pass + @pytest.mark.asyncio -async def test_call_tool_missing_path_parameter(http_transport, logger): +async def test_call_tool_missing_path_parameter(http_transport): """Test error handling when path parameters are missing.""" - # Create a provider with path parameters - provider = HttpProvider( - name="test_provider", + # Create a call template with path parameters + call_template = HttpCallTemplate( + name="test_call_template", url="https://api.example.com/users/{user_id}/posts/{post_id}", http_method="GET" ) @@ -511,31 +665,32 @@ async def test_call_tool_missing_path_parameter(http_transport, logger): # Try to call the tool without required path parameters with pytest.raises(ValueError, match="Missing required path parameter: post_id"): await http_transport.call_tool( + None, "test_tool", {"user_id": "123"}, # Missing post_id - provider + call_template ) @pytest.mark.asyncio -async def test_call_tool_openlibrary_style_url(http_transport, logger): +async def test_call_tool_openlibrary_style_url(http_transport): """Test calling a tool with OpenLibrary-style URL path parameters.""" - # Create a provider with OpenLibrary-style URL (the original problem case) - provider = HttpProvider( - name="openlibrary_provider", + # Create a call template with OpenLibrary-style URL (the original problem case) + call_template = HttpCallTemplate( + name="openlibrary_call_template", url="https://openlibrary.org/api/volumes/brief/{key_type}/{value}.json", http_method="GET" ) # Test the URL building (we can't make actual requests to OpenLibrary in tests) arguments = {"key_type": "isbn", "value": "9780140328721", "format": "json"} - url = http_transport._build_url_with_path_params(provider.url, arguments.copy()) + url = http_transport._build_url_with_path_params(call_template.url, arguments.copy()) # Verify the URL was built correctly assert url == "https://openlibrary.org/api/volumes/brief/isbn/9780140328721.json" # Verify that path parameters were removed from arguments, leaving only query parameters expected_remaining = {"format": "json"} - http_transport._build_url_with_path_params(provider.url, arguments) + http_transport._build_url_with_path_params(call_template.url, arguments) assert arguments == expected_remaining diff --git a/tests/client/test_openapi_converter.py b/plugins/communication_protocols/http/tests/test_openapi_converter.py similarity index 66% rename from tests/client/test_openapi_converter.py rename to plugins/communication_protocols/http/tests/test_openapi_converter.py index 6bee599..77382c7 100644 --- a/tests/client/test_openapi_converter.py +++ b/plugins/communication_protocols/http/tests/test_openapi_converter.py @@ -1,8 +1,8 @@ import pytest import aiohttp import sys -from utcp.client.openapi_converter import OpenApiConverter -from utcp.shared.utcp_manual import UtcpManual +from utcp_http.openapi_converter import OpenApiConverter +from utcp.data.utcp_manual import UtcpManual @pytest.mark.asyncio @@ -24,8 +24,11 @@ async def test_openai_spec_conversion(): # Check a few things on a sample tool to ensure parsing is reasonable sample_tool = next((tool for tool in utcp_manual.tools if tool.name == "createChatCompletion"), None) assert sample_tool is not None - assert sample_tool.tool_provider.provider_type == "http" - assert sample_tool.tool_provider.http_method == "POST" - assert "messages" in sample_tool.inputs.properties['body']['properties'] - assert "model" in sample_tool.inputs.properties['body']['properties'] + assert sample_tool.tool_call_template.call_template_type == "http" + assert sample_tool.tool_call_template.http_method == "POST" + body_schema = sample_tool.inputs.properties.get('body') + assert body_schema is not None + assert body_schema.properties is not None + assert "messages" in body_schema.properties + assert "model" in body_schema.properties assert "choices" in sample_tool.outputs.properties diff --git a/tests/client/test_openapi_converter_auth.py b/plugins/communication_protocols/http/tests/test_openapi_converter_auth.py similarity index 74% rename from tests/client/test_openapi_converter_auth.py rename to plugins/communication_protocols/http/tests/test_openapi_converter_auth.py index a30a498..29a51ff 100644 --- a/tests/client/test_openapi_converter_auth.py +++ b/plugins/communication_protocols/http/tests/test_openapi_converter_auth.py @@ -1,9 +1,9 @@ import pytest import aiohttp -from utcp.client.openapi_converter import OpenApiConverter -from utcp.shared.utcp_manual import UtcpManual -from utcp.shared.auth import ApiKeyAuth -from utcp.shared.provider import HttpProvider +from utcp_http.openapi_converter import OpenApiConverter +from utcp.data.utcp_manual import UtcpManual +from utcp.data.auth_implementations import ApiKeyAuth +from utcp_http.http_call_template import HttpCallTemplate @pytest.mark.asyncio @@ -22,11 +22,11 @@ async def test_webscraping_ai_spec_conversion(): assert isinstance(utcp_manual, UtcpManual) assert len(utcp_manual.tools) == 4 # account, getHTML, getSelected, getSelectedMultiple - # Check that all tools are HTTP providers + # Check that all tools use HTTP call templates for tool in utcp_manual.tools: - assert isinstance(tool.tool_provider, HttpProvider) - assert tool.tool_provider.provider_type == "http" - assert tool.tool_provider.http_method == "GET" + assert isinstance(tool.tool_call_template, HttpCallTemplate) + assert tool.tool_call_template.call_template_type == "http" + assert tool.tool_call_template.http_method == "GET" @pytest.mark.asyncio @@ -44,11 +44,11 @@ async def test_webscraping_ai_auth_extraction(): # All tools should have API key authentication for tool in utcp_manual.tools: - assert tool.tool_provider.auth is not None - assert isinstance(tool.tool_provider.auth, ApiKeyAuth) - assert tool.tool_provider.auth.var_name == "api_key" - assert tool.tool_provider.auth.api_key.startswith("${API_KEY_") - assert tool.tool_provider.auth.location == "query" + assert tool.tool_call_template.auth is not None + assert isinstance(tool.tool_call_template.auth, ApiKeyAuth) + assert tool.tool_call_template.auth.var_name == "api_key" + assert tool.tool_call_template.auth.api_key.startswith("${API_KEY_") + assert tool.tool_call_template.auth.location == "query" @pytest.mark.asyncio @@ -68,14 +68,14 @@ async def test_webscraping_ai_specific_tools(): account_tool = next((tool for tool in utcp_manual.tools if tool.name == "account"), None) assert account_tool is not None assert account_tool.description == "Information about your account calls quota" - assert account_tool.tool_provider.url == "https://api.webscraping.ai/account" + assert account_tool.tool_call_template.url == "https://api.webscraping.ai/account" assert "Account" in account_tool.tags # Test getHTML tool html_tool = next((tool for tool in utcp_manual.tools if tool.name == "getHTML"), None) assert html_tool is not None assert html_tool.description == "Page HTML by URL" - assert html_tool.tool_provider.url == "https://api.webscraping.ai/html" + assert html_tool.tool_call_template.url == "https://api.webscraping.ai/html" assert "HTML" in html_tool.tags # Check that URL parameter is required @@ -86,16 +86,16 @@ async def test_webscraping_ai_specific_tools(): # Test getSelected tool selected_tool = next((tool for tool in utcp_manual.tools if tool.name == "getSelected"), None) assert selected_tool is not None - assert selected_tool.tool_provider.url == "https://api.webscraping.ai/selected" + assert selected_tool.tool_call_template.url == "https://api.webscraping.ai/selected" assert "selector" in selected_tool.inputs.properties assert "url" in selected_tool.inputs.properties # Test getSelectedMultiple tool selected_multiple_tool = next((tool for tool in utcp_manual.tools if tool.name == "getSelectedMultiple"), None) assert selected_multiple_tool is not None - assert selected_multiple_tool.tool_provider.url == "https://api.webscraping.ai/selected-multiple" + assert selected_multiple_tool.tool_call_template.url == "https://api.webscraping.ai/selected-multiple" assert "selectors" in selected_multiple_tool.inputs.properties - assert selected_multiple_tool.inputs.properties["selectors"]["type"] == "array" + assert selected_multiple_tool.inputs.properties["selectors"].type == "array" @pytest.mark.asyncio @@ -117,17 +117,23 @@ async def test_webscraping_ai_parameter_resolution(): # Check that referenced parameters are properly resolved assert "url" in html_tool.inputs.properties - assert html_tool.inputs.properties["url"]["description"] == "URL of the target page" - assert html_tool.inputs.properties["url"]["type"] == "string" - + url_schema = html_tool.inputs.properties.get("url") + assert url_schema is not None + assert url_schema.description == "URL of the target page" + assert url_schema.type == "string" + assert "timeout" in html_tool.inputs.properties - assert html_tool.inputs.properties["timeout"]["description"].startswith("Maximum processing time in ms") - assert html_tool.inputs.properties["timeout"]["type"] == "integer" - assert html_tool.inputs.properties["timeout"]["default"] == 10000 - + timeout_schema = html_tool.inputs.properties.get("timeout") + assert timeout_schema is not None + assert isinstance(timeout_schema.description, str) and timeout_schema.description.startswith("Maximum processing time in ms") + assert timeout_schema.type == "integer" + assert timeout_schema.default == 10000 + assert "js" in html_tool.inputs.properties - assert html_tool.inputs.properties["js"]["type"] == "boolean" - assert html_tool.inputs.properties["js"]["default"] is True + js_schema = html_tool.inputs.properties.get("js") + assert js_schema is not None + assert js_schema.type == "boolean" + assert js_schema.default is True @pytest.mark.asyncio @@ -154,7 +160,7 @@ async def test_webscraping_ai_response_schemas(): # Test getHTML tool output schema (should be string for HTML) html_tool = next((tool for tool in utcp_manual.tools if tool.name == "getHTML"), None) assert html_tool is not None - assert html_tool.outputs.type == "object" + assert html_tool.outputs.type == "string" # Test getSelectedMultiple tool output schema (should be array) selected_multiple_tool = next((tool for tool in utcp_manual.tools if tool.name == "getSelectedMultiple"), None) @@ -162,4 +168,4 @@ async def test_webscraping_ai_response_schemas(): assert selected_multiple_tool.outputs.type == "array" # Now we can check array item types with our enhanced schema assert selected_multiple_tool.outputs.items is not None - assert selected_multiple_tool.outputs.items.get("type") == "string" + assert selected_multiple_tool.outputs.items.type == "string" diff --git a/plugins/communication_protocols/http/tests/test_sse_communication_protocol.py b/plugins/communication_protocols/http/tests/test_sse_communication_protocol.py new file mode 100644 index 0000000..2c4a2eb --- /dev/null +++ b/plugins/communication_protocols/http/tests/test_sse_communication_protocol.py @@ -0,0 +1,400 @@ +import pytest +import pytest_asyncio +import json +import asyncio +import base64 +from unittest.mock import MagicMock, patch, AsyncMock + +import aiohttp +from aiohttp import web + +from utcp_http.sse_communication_protocol import SseCommunicationProtocol +from utcp_http.sse_call_template import SseCallTemplate +from utcp.data.auth_implementations import ApiKeyAuth, BasicAuth, OAuth2Auth +from utcp.data.register_manual_response import RegisterManualResult + +# --- Test Data --- + +SAMPLE_SSE_EVENTS = [ + 'id: 1\ndata: {"message": "First part"}\n\n', + 'id: 2\nevent: data\ndata: { "message": "Second part" }\n\n', + 'id: 3\nevent: complete\ndata: { "message": "End of stream" }\n\n' +] + +# --- Test Server Handlers --- + +async def tools_handler(request): + execution_call_template = { + "call_template_type": "sse", + "name": "test-sse-call-template-executor", + "url": str(request.url.origin()) + "/events", + "http_method": "GET", + "content_type": "application/json" + } + utcp_manual = { + "utcp_version": "1.0.0", + "manual_version": "1.0.0", + "tools": [ + { + "name": "test_tool", + "description": "Test tool", + "inputs": { + "type": "object", + "properties": {"param1": {"type": "string"}} + }, + "outputs": { + "type": "object", + "properties": {"result": {"type": "string"}} + }, + "tags": [], + "tool_call_template": execution_call_template + } + ] + } + return web.json_response(utcp_manual) + +async def events_handler(request): + if request.method not in ('GET', 'POST'): + return web.Response(status=405) + + # Check auth + if 'X-API-Key' in request.headers and request.headers['X-API-Key'] != 'test-api-key': + return web.Response(status=401, text="Invalid API Key") + if 'Authorization' in request.headers: + auth_header = request.headers['Authorization'] + if auth_header.startswith('Basic'): + if auth_header != f"Basic {base64.b64encode(b'user:pass').decode()}": + return web.Response(status=401, text="Invalid Basic Auth") + elif auth_header.startswith('Bearer'): + if auth_header not in ('Bearer test-access-token', 'Bearer test-access-token-header'): + return web.Response(status=401, text="Invalid Bearer Token") + + response = web.StreamResponse( + status=200, + reason='OK', + headers={'Content-Type': 'text/event-stream'} + ) + await response.prepare(request) + + for event in SAMPLE_SSE_EVENTS: + await response.write(event.encode('utf-8')) + await asyncio.sleep(0.01) # Simulate network delay + + return response + +async def token_handler(request): + data = await request.post() + if data.get('client_id') == 'client-id' and data.get('client_secret') == 'client-secret': + return web.json_response({ + "access_token": "test-access-token", + "token_type": "Bearer", + "expires_in": 3600 + }) + return web.json_response({"error": "invalid_client"}, status=401) + +async def token_header_auth_handler(request): + auth_header = request.headers.get('Authorization') + if auth_header == f"Basic {base64.b64encode(b'client-id:client-secret').decode()}": + return web.json_response({ + "access_token": "test-access-token-header", + "token_type": "Bearer", + "expires_in": 3600 + }) + return web.json_response({"error": "invalid_client"}, status=401) + +async def error_handler(request): + return web.Response(status=500, text="Internal Server Error") + +# --- Pytest Fixtures --- + +@pytest.fixture +def sse_transport(): + """Fixture to create and properly tear down an SseCommunicationProtocol instance.""" + transport = SseCommunicationProtocol() + yield transport + asyncio.run(transport.close()) + +@pytest.fixture +def app(): + app = web.Application() + app.router.add_get("/tools", tools_handler) + app.router.add_route('*', '/events', events_handler) + app.router.add_post("/token", token_handler) + app.router.add_post("/token_header_auth", token_header_auth_handler) + app.router.add_get("/error", error_handler) + return app + +@pytest_asyncio.fixture +async def oauth2_call_template(aiohttp_client, app): + client = await aiohttp_client(app) + return SseCallTemplate( + name="oauth2-call-template", + url=f"{client.make_url('/events')}", + auth=OAuth2Auth( + client_id="client-id", + client_secret="client-secret", + token_url=f"{client.make_url('/token')}", + scope="read write" + ) + ) + +# --- Tests --- + +@pytest.mark.asyncio +async def test_register_manual(sse_transport, aiohttp_client, app): + """Test registering a manual.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate(name="test-call-template", url=f"{client.make_url('/tools')}") + result = await sse_transport.register_manual(None, call_template) + + assert isinstance(result, RegisterManualResult) + assert result.success + assert not result.errors + assert result.manual is not None + assert len(result.manual.tools) == 1 + assert result.manual.tools[0].name == "test_tool" + +@pytest.mark.asyncio +async def test_register_manual_error(sse_transport, aiohttp_client, app): + """Test error handling when registering a manual.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate(name="test-error", url=f"{client.make_url('/error')}") + result = await sse_transport.register_manual(None, call_template) + assert not result.success + assert result.manual is not None + assert len(result.manual.tools) == 0 + assert result.errors + assert isinstance(result.errors[0], str) + +@pytest.mark.asyncio +async def test_call_tool_basic(sse_transport, aiohttp_client, app): + """Test calling a tool with basic configuration.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate(name="test-basic", url=f"{client.make_url('/events')}") + + events = [] + async for event in sse_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, call_template): + events.append(event) + + assert len(events) == 3 + assert events[0] == {"message": "First part"} + assert events[1] == {"message": "Second part"} + assert events[2] == {"message": "End of stream"} + +@pytest.mark.asyncio +async def test_call_tool_with_api_key(sse_transport, aiohttp_client, app): + """Test calling a tool with API key authentication.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate( + name="api-key-call-template", + url=f"{client.make_url('/events')}", + auth=ApiKeyAuth(api_key="test-api-key", header_name="X-API-Key") + ) + stream_iterator = sse_transport.call_tool_streaming(None, "test_tool", {}, call_template) + results = [event async for event in stream_iterator] + assert len(results) == 3 + +@pytest.mark.asyncio +async def test_call_tool_with_basic_auth(sse_transport, aiohttp_client, app): + """Test calling a tool with Basic authentication.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate( + name="basic-auth-call-template", + url=f"{client.make_url('/events')}", + auth=BasicAuth(username="user", password="pass") + ) + stream_iterator = sse_transport.call_tool_streaming(None, "test_tool", {}, call_template) + results = [event async for event in stream_iterator] + assert len(results) == 3 + +@pytest.mark.asyncio +async def test_call_tool_with_oauth2(sse_transport, oauth2_call_template, app): + """Test calling a tool with OAuth2 authentication (credentials in body).""" + events = [] + async for event in sse_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, oauth2_call_template): + events.append(event) + + assert len(events) == 3 + assert events[0] == {"message": "First part"} + assert events[1] == {"message": "Second part"} + assert events[2] == {"message": "End of stream"} + +@pytest.mark.asyncio +async def test_call_tool_with_oauth2_header_auth(sse_transport, aiohttp_client, app): + """Test calling a tool with OAuth2 authentication (credentials in header).""" + client = await aiohttp_client(app) + oauth2_header_call_template = SseCallTemplate( + name="oauth2-header-call-template", + url=f"{client.make_url('/events')}", + auth=OAuth2Auth( + client_id="client-id", + client_secret="client-secret", + token_url=f"{client.make_url('/token_header_auth')}", + scope="read write" + ) + ) + + events = [] + async for event in sse_transport.call_tool_streaming(None, "test_tool", {"param1": "value1"}, oauth2_header_call_template): + events.append(event) + + assert len(events) == 3 + assert events[0] == {"message": "First part"} + assert events[1] == {"message": "Second part"} + assert events[2] == {"message": "End of stream"} + +@pytest.mark.asyncio +async def test_call_tool_with_body_field(sse_transport, aiohttp_client, app): + """Test calling a tool with a body field.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate( + name="body-field-call-template", + url=f"{client.make_url('/events')}", + body_field="data", + headers={"Content-Type": "application/json"} + ) + stream_iterator = sse_transport.call_tool_streaming( + None, + "test_tool", + {"param1": "value1", "data": {"key": "value"}}, + call_template + ) + results = [event async for event in stream_iterator] + assert len(results) == 3 + +@pytest.mark.asyncio +async def test_call_tool_error(sse_transport, aiohttp_client, app): + """Test error handling when calling a tool.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate(name="test-error", url=f"{client.make_url('/error')}") + with pytest.raises(aiohttp.ClientResponseError) as excinfo: + async for _ in sse_transport.call_tool_streaming(None, "test_tool", {}, call_template): + pass + + assert excinfo.value.status == 500 + +@pytest.mark.asyncio +async def test_deregister_manual(sse_transport, aiohttp_client, app): + """Test deregistering a manual closes the connection.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate(name="test-deregister", url=f"{client.make_url('/events')}") + + # Make a call to establish a connection + stream_iterator = sse_transport.call_tool_streaming(None, "test_tool", {}, call_template) + await anext(stream_iterator) + assert call_template.name in sse_transport._active_connections + response, session = sse_transport._active_connections[call_template.name] + + # Deregister + await sse_transport.deregister_manual(None, call_template) + + # Verify connection and session are closed and removed + assert call_template.name not in sse_transport._active_connections + assert response.closed + assert session.closed + +@pytest.mark.asyncio +async def test_call_tool_basic_nonstream(sse_transport, aiohttp_client, app): + """Non-streaming call should aggregate SSE events into a list (basic).""" + client = await aiohttp_client(app) + call_template = SseCallTemplate(name="test-basic", url=f"{client.make_url('/events')}") + + result = await sse_transport.call_tool(None, "test_tool", {"param1": "value1"}, call_template) + + assert isinstance(result, list) + assert len(result) == 3 + assert result[0] == {"message": "First part"} + assert result[1] == {"message": "Second part"} + assert result[2] == {"message": "End of stream"} + +@pytest.mark.asyncio +async def test_call_tool_with_api_key_nonstream(sse_transport, aiohttp_client, app): + """Non-streaming call with API key should behave like streaming.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate( + name="api-key-call-template", + url=f"{client.make_url('/events')}", + auth=ApiKeyAuth(api_key="test-api-key", header_name="X-API-Key") + ) + + result = await sse_transport.call_tool(None, "test_tool", {}, call_template) + + assert isinstance(result, list) + assert len(result) == 3 + +@pytest.mark.asyncio +async def test_call_tool_with_basic_auth_nonstream(sse_transport, aiohttp_client, app): + """Non-streaming call with Basic auth should behave like streaming.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate( + name="basic-auth-call-template", + url=f"{client.make_url('/events')}", + auth=BasicAuth(username="user", password="pass") + ) + + result = await sse_transport.call_tool(None, "test_tool", {}, call_template) + + assert isinstance(result, list) + assert len(result) == 3 + +@pytest.mark.asyncio +async def test_call_tool_with_oauth2_nonstream(sse_transport, oauth2_call_template, app): + """Non-streaming call with OAuth2 (body credentials) should aggregate events.""" + result = await sse_transport.call_tool(None, "test_tool", {"param1": "value1"}, oauth2_call_template) + assert isinstance(result, list) + assert len(result) == 3 + assert result[0] == {"message": "First part"} + assert result[1] == {"message": "Second part"} + assert result[2] == {"message": "End of stream"} + +@pytest.mark.asyncio +async def test_call_tool_with_oauth2_header_auth_nonstream(sse_transport, aiohttp_client, app): + """Non-streaming call with OAuth2 (header credentials) should aggregate events.""" + client = await aiohttp_client(app) + oauth2_header_call_template = SseCallTemplate( + name="oauth2-header-call-template", + url=f"{client.make_url('/events')}", + auth=OAuth2Auth( + client_id="client-id", + client_secret="client-secret", + token_url=f"{client.make_url('/token_header_auth')}", + scope="read write" + ) + ) + + result = await sse_transport.call_tool(None, "test_tool", {"param1": "value1"}, oauth2_header_call_template) + + assert isinstance(result, list) + assert len(result) == 3 + assert result[0] == {"message": "First part"} + assert result[1] == {"message": "Second part"} + assert result[2] == {"message": "End of stream"} + +@pytest.mark.asyncio +async def test_call_tool_with_body_field_nonstream(sse_transport, aiohttp_client, app): + """Non-streaming call with body field should aggregate events.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate( + name="body-field-call-template", + url=f"{client.make_url('/events')}", + body_field="data", + headers={"Content-Type": "application/json"} + ) + + result = await sse_transport.call_tool( + None, + "test_tool", + {"param1": "value1", "data": {"key": "value"}}, + call_template + ) + assert isinstance(result, list) + assert len(result) == 3 + +@pytest.mark.asyncio +async def test_call_tool_error_nonstream(sse_transport, aiohttp_client, app): + """Non-streaming call should raise same error on server failure.""" + client = await aiohttp_client(app) + call_template = SseCallTemplate(name="test-error", url=f"{client.make_url('/error')}") + with pytest.raises(aiohttp.ClientResponseError) as excinfo: + await sse_transport.call_tool(None, "test_tool", {}, call_template) + assert excinfo.value.status == 500 diff --git a/plugins/communication_protocols/http/tests/test_streamable_http_communication_protocol.py b/plugins/communication_protocols/http/tests/test_streamable_http_communication_protocol.py new file mode 100644 index 0000000..d86a44c --- /dev/null +++ b/plugins/communication_protocols/http/tests/test_streamable_http_communication_protocol.py @@ -0,0 +1,343 @@ +import pytest +import pytest_asyncio +import json +import asyncio +import aiohttp +from aiohttp import web + +from utcp_http.streamable_http_communication_protocol import StreamableHttpCommunicationProtocol +from utcp_http.streamable_http_call_template import StreamableHttpCallTemplate +from utcp.data.auth_implementations import ApiKeyAuth, BasicAuth, OAuth2Auth +from utcp.data.register_manual_response import RegisterManualResult + +# --- Test Data --- + +SAMPLE_NDJSON_RESPONSE = [ + {'status': 'running', 'progress': 0}, + {'status': 'running', 'progress': 50}, + {'status': 'completed', 'result': 'done'} +] + +# --- Fixtures --- + +@pytest_asyncio.fixture +async def streamable_http_transport(): + """Fixture to create and properly tear down a StreamableHttpCommunicationProtocol instance.""" + transport = StreamableHttpCommunicationProtocol() + yield transport + await transport.close() + +@pytest.fixture +def app(): + """Fixture for the aiohttp test application.""" + async def discover(request): + execution_call_template = { + "call_template_type": "streamable_http", + "name": "test-streamable-http-executor", + "url": str(request.url.origin()) + "/stream-ndjson", + "http_method": "GET", + "content_type": "application/x-ndjson" + } + utcp_manual = { + "utcp_version": "1.0.0", + "manual_version": "1.0.0", + "tools": [ + { + "name": "test_tool", + "description": "Test tool", + "inputs": {}, + "outputs": {}, + "tags": [], + "tool_call_template": execution_call_template + } + ] + } + return web.json_response(utcp_manual) + + async def stream_ndjson(request): + response = web.StreamResponse( + status=200, + reason='OK', + headers={'Content-Type': 'application/x-ndjson'} + ) + await response.prepare(request) + for item in SAMPLE_NDJSON_RESPONSE: + await response.write(json.dumps(item).encode('utf-8') + b'\n') + await asyncio.sleep(0.01) # Simulate network delay + return response + + async def stream_binary(request): + response = web.StreamResponse( + status=200, + reason='OK', + headers={'Content-Type': 'application/octet-stream'} + ) + await response.prepare(request) + await response.write(b'chunk1') + await response.write(b'chunk2') + return response + + async def check_api_key_auth(request): + if request.headers.get("X-API-Key") != "test-key": + return web.Response(status=401, text="Unauthorized: Invalid API Key") + return await stream_ndjson(request) + + async def check_basic_auth(request): + auth_header = request.headers.get('Authorization') + if not auth_header or 'Basic dXNlcjpwYXNz' not in auth_header: # user:pass + return web.Response(status=401, text="Unauthorized: Invalid Basic Auth") + return await stream_ndjson(request) + + async def oauth_token_handler(request): + data = await request.post() + if data.get('client_id') == 'test-client' and data.get('client_secret') == 'test-secret': + return web.json_response({'access_token': 'token-from-body', 'token_type': 'Bearer'}) + return web.Response(status=401, text="Invalid client credentials") + + async def oauth_token_header_handler(request): + auth_header = request.headers.get('Authorization') + if auth_header and 'Basic dGVzdC1jbGllbnQ6dGVzdC1zZWNyZXQ=' in auth_header: # test-client:test-secret + return web.json_response({'access_token': 'token-from-header', 'token_type': 'Bearer'}) + return web.Response(status=401, text="Invalid client credentials via header") + + async def check_oauth(request): + auth_header = request.headers.get('Authorization') + if auth_header in ('Bearer token-from-body', 'Bearer token-from-header'): + return await stream_ndjson(request) + return web.Response(status=401, text="Unauthorized: Invalid OAuth Token") + + async def error_endpoint(request): + return web.Response(status=500, text="Internal Server Error") + + app = web.Application() + app.add_routes([ + web.get('/discover', discover), + web.get('/stream-ndjson', stream_ndjson), + web.get('/stream-binary', stream_binary), + web.get('/auth-api-key', check_api_key_auth), + web.get('/auth-basic', check_basic_auth), + web.get('/auth-oauth', check_oauth), + web.post('/token', oauth_token_handler), + web.post('/token-header', oauth_token_header_handler), + web.get('/error', error_endpoint), + ]) + return app + +# --- Test Cases --- + +@pytest.mark.asyncio +async def test_register_manual(streamable_http_transport, aiohttp_client, app): + """Test successful manual registration.""" + client = await aiohttp_client(app) + call_template = StreamableHttpCallTemplate(name="test-provider", url=f"{client.make_url('/discover')}") + result = await streamable_http_transport.register_manual(None, call_template) + + assert isinstance(result, RegisterManualResult) + assert result.success + assert not result.errors + assert result.manual is not None + assert len(result.manual.tools) == 1 + assert result.manual.tools[0].name == "test_tool" + +@pytest.mark.asyncio +async def test_register_manual_error(streamable_http_transport, aiohttp_client, app): + """Test error handling during manual registration.""" + client = await aiohttp_client(app) + call_template = StreamableHttpCallTemplate(name="test-provider", url=f"{client.make_url('/error')}") + result = await streamable_http_transport.register_manual(None, call_template) + + assert isinstance(result, RegisterManualResult) + assert not result.success + assert result.errors + assert isinstance(result.errors[0], str) + assert result.manual is not None + assert len(result.manual.tools) == 0 + +@pytest.mark.asyncio +async def test_call_tool_streaming_ndjson(streamable_http_transport, aiohttp_client, app): + """Test calling a tool that returns an NDJSON stream.""" + client = await aiohttp_client(app) + call_template = StreamableHttpCallTemplate(name="ndjson-provider", url=f"{client.make_url('/stream-ndjson')}", content_type='application/x-ndjson') + + stream_iterator = streamable_http_transport.call_tool_streaming( + None, "test_tool", {}, call_template + ) + + results = [item async for item in stream_iterator] + + assert results == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_binary_stream(streamable_http_transport, aiohttp_client, app): + """Test calling a tool that returns a binary stream.""" + client = await aiohttp_client(app) + call_template = StreamableHttpCallTemplate( + name="binary-provider", + url=f"{client.make_url('/stream-binary')}", + content_type='application/octet-stream', + chunk_size=6 + ) + + stream_iterator = streamable_http_transport.call_tool_streaming(None, "test_tool", {}, call_template) + + results = [chunk async for chunk in stream_iterator] + + assert results == [b'chunk1', b'chunk2'] + +@pytest.mark.asyncio +async def test_call_tool_with_api_key(streamable_http_transport, aiohttp_client, app): + """Test that the API key is correctly sent in the headers.""" + client = await aiohttp_client(app) + auth = ApiKeyAuth(var_name="X-API-Key", api_key="test-key", location="header") + call_template = StreamableHttpCallTemplate( + name="auth-provider", + url=f"{client.make_url('/auth-api-key')}", + auth=auth, + content_type='application/x-ndjson' + ) + + stream_iterator = streamable_http_transport.call_tool_streaming(None, "test_tool", {}, call_template) + results = [item async for item in stream_iterator] + + assert results == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_with_basic_auth(streamable_http_transport, aiohttp_client, app): + """Test streaming with Basic authentication.""" + client = await aiohttp_client(app) + auth = BasicAuth(username="user", password="pass") + call_template = StreamableHttpCallTemplate( + name="basic-auth-provider", + url=f"{client.make_url('/auth-basic')}", + auth=auth, + content_type='application/x-ndjson' + ) + + stream_iterator = streamable_http_transport.call_tool_streaming(None, "test_tool", {}, call_template) + results = [item async for item in stream_iterator] + + assert results == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_with_oauth2_body(streamable_http_transport, aiohttp_client, app): + """Test streaming with OAuth2 (credentials in body).""" + client = await aiohttp_client(app) + auth = OAuth2Auth(client_id="test-client", client_secret="test-secret", token_url=f"{client.make_url('/token')}") + call_template = StreamableHttpCallTemplate( + name="oauth-provider", + url=f"{client.make_url('/auth-oauth')}", + auth=auth, + content_type='application/x-ndjson' + ) + + stream_iterator = streamable_http_transport.call_tool_streaming(None, "test_tool", {}, call_template) + results = [item async for item in stream_iterator] + + assert results == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_with_oauth2_header_fallback(streamable_http_transport, aiohttp_client, app): + """Test streaming with OAuth2 (fallback to Basic Auth header).""" + client = await aiohttp_client(app) + # This token endpoint will fail for the body method, forcing a fallback. + auth = OAuth2Auth(client_id="test-client", client_secret="test-secret", token_url=f"{client.make_url('/token-header')}") + call_template = StreamableHttpCallTemplate( + name="oauth-fallback-provider", + url=f"{client.make_url('/auth-oauth')}", + auth=auth, + content_type='application/x-ndjson' + ) + + stream_iterator = streamable_http_transport.call_tool_streaming(None, "test_tool", {}, call_template) + results = [item async for item in stream_iterator] + + assert results == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_ndjson(streamable_http_transport, aiohttp_client, app): + """Non-streaming call should return full list for NDJSON.""" + client = await aiohttp_client(app) + call_template = StreamableHttpCallTemplate(name="ndjson-provider", url=f"{client.make_url('/stream-ndjson')}", content_type='application/x-ndjson') + + result = await streamable_http_transport.call_tool(None, "test_tool", {}, call_template) + + assert result == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_binary(streamable_http_transport, aiohttp_client, app): + """Non-streaming call should return concatenated bytes for binary stream.""" + client = await aiohttp_client(app) + call_template = StreamableHttpCallTemplate( + name="binary-provider", + url=f"{client.make_url('/stream-binary')}", + content_type='application/octet-stream', + chunk_size=6 + ) + + result = await streamable_http_transport.call_tool(None, "test_tool", {}, call_template) + + assert result == b'chunk1chunk2' + +@pytest.mark.asyncio +async def test_call_tool_with_api_key_nonstream(streamable_http_transport, aiohttp_client, app): + """Non-streaming call with API key in header should behave like streaming.""" + client = await aiohttp_client(app) + auth = ApiKeyAuth(var_name="X-API-Key", api_key="test-key", location="header") + call_template = StreamableHttpCallTemplate( + name="auth-provider", + url=f"{client.make_url('/auth-api-key')}", + auth=auth, + content_type='application/x-ndjson' + ) + + result = await streamable_http_transport.call_tool(None, "test_tool", {}, call_template) + + assert result == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_with_basic_auth_nonstream(streamable_http_transport, aiohttp_client, app): + """Non-streaming call with Basic auth should behave like streaming.""" + client = await aiohttp_client(app) + auth = BasicAuth(username="user", password="pass") + call_template = StreamableHttpCallTemplate( + name="basic-auth-provider", + url=f"{client.make_url('/auth-basic')}", + auth=auth, + content_type='application/x-ndjson' + ) + + result = await streamable_http_transport.call_tool(None, "test_tool", {}, call_template) + + assert result == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_with_oauth2_body_nonstream(streamable_http_transport, aiohttp_client, app): + """Non-streaming call with OAuth2 (credentials in body) should behave like streaming.""" + client = await aiohttp_client(app) + auth = OAuth2Auth(client_id="test-client", client_secret="test-secret", token_url=f"{client.make_url('/token')}") + call_template = StreamableHttpCallTemplate( + name="oauth-provider", + url=f"{client.make_url('/auth-oauth')}", + auth=auth, + content_type='application/x-ndjson' + ) + + result = await streamable_http_transport.call_tool(None, "test_tool", {}, call_template) + + assert result == SAMPLE_NDJSON_RESPONSE + +@pytest.mark.asyncio +async def test_call_tool_with_oauth2_header_fallback_nonstream(streamable_http_transport, aiohttp_client, app): + """Non-streaming call with OAuth2 (fallback to Basic Auth header) should behave like streaming.""" + client = await aiohttp_client(app) + auth = OAuth2Auth(client_id="test-client", client_secret="test-secret", token_url=f"{client.make_url('/token-header')}") + call_template = StreamableHttpCallTemplate( + name="oauth-fallback-provider", + url=f"{client.make_url('/auth-oauth')}", + auth=auth, + content_type='application/x-ndjson' + ) + + result = await streamable_http_transport.call_tool(None, "test_tool", {}, call_template) + + assert result == SAMPLE_NDJSON_RESPONSE diff --git a/plugins/communication_protocols/mcp/pyproject.toml b/plugins/communication_protocols/mcp/pyproject.toml new file mode 100644 index 0000000..8d243f7 --- /dev/null +++ b/plugins/communication_protocols/mcp/pyproject.toml @@ -0,0 +1,43 @@ +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" + +[project] +name = "utcp-mcp" +version = "1.0.0" +authors = [ + { name = "UTCP Contributors" }, +] +description = "Universal Tool Calling Protocol (UTCP) client library for Python" +readme = "README.md" +requires-python = ">=3.10" +dependencies = [ + "pydantic>=2.0", + "mcp>=1.12", + "utcp>=1.0" +] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "Programming Language :: Python :: 3", + "Operating System :: OS Independent", +] +license = "MPL-2.0" + +[project.optional-dependencies] +dev = [ + "build", + "pytest", + "pytest-asyncio", + "pytest-cov", + "coverage", + "twine", +] + +[project.urls] +Homepage = "https://utcp.io" +Source = "https://github.com/universal-tool-calling-protocol/python-utcp" +Issues = "https://github.com/universal-tool-calling-protocol/python-utcp/issues" + +[project.entry-points."utcp.plugins"] +mcp = "utcp_mcp:register" \ No newline at end of file diff --git a/plugins/communication_protocols/mcp/src/utcp_mcp/__init__.py b/plugins/communication_protocols/mcp/src/utcp_mcp/__init__.py new file mode 100644 index 0000000..85abb78 --- /dev/null +++ b/plugins/communication_protocols/mcp/src/utcp_mcp/__init__.py @@ -0,0 +1,13 @@ +from utcp_mcp.mcp_communication_protocol import McpCommunicationProtocol +from utcp_mcp.mcp_call_template import McpCallTemplate, McpCallTemplateSerializer +from utcp.plugins.discovery import register_communication_protocol, register_call_template + +def register(): + register_communication_protocol("mcp", McpCommunicationProtocol()) + register_call_template("mcp", McpCallTemplateSerializer()) + +__all__ = [ + "McpCommunicationProtocol", + "McpCallTemplate", + "McpCallTemplateSerializer", +] diff --git a/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.py b/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.py new file mode 100644 index 0000000..c62901f --- /dev/null +++ b/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.py @@ -0,0 +1,54 @@ + +from pydantic import BaseModel +from typing import Optional, Dict, Literal, Any +from utcp.data.auth_implementations import OAuth2Auth +from utcp.data.call_template import CallTemplate +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +"""Type alias for MCP server configurations. + +Union type for all supported MCP server transport configurations, +including both stdio and HTTP-based servers. +""" + +class McpConfig(BaseModel): + """Configuration container for multiple MCP servers. + + Holds a collection of named MCP server configurations, allowing + a single MCP provider to manage multiple server connections. + + Attributes: + mcpServers: Dictionary mapping server names to their configurations. + """ + + mcpServers: Dict[str, Dict[str, Any]] + +class McpCallTemplate(CallTemplate): + """Provider configuration for Model Context Protocol (MCP) tools. + + Enables communication with MCP servers that provide structured tool + interfaces. Supports both stdio (local process) and HTTP (remote) + transport methods. + + Attributes: + call_template_type: Always "mcp" for MCP providers. + config: Configuration object containing MCP server definitions. + This follows the same format as the official MCP server configuration. + auth: Optional OAuth2 authentication for HTTP-based MCP servers. + """ + + call_template_type: Literal["mcp"] = "mcp" + config: McpConfig + auth: Optional[OAuth2Auth] = None + +class McpCallTemplateSerializer(Serializer[McpCallTemplate]): + def to_dict(self, obj: McpCallTemplate) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> McpCallTemplate: + try: + return McpCallTemplate.model_validate(obj) + except Exception as e: + raise UtcpSerializerValidationError("Invalid McpCallTemplate: " + traceback.format_exc()) from e diff --git a/src/utcp/client/transport_interfaces/mcp_transport.py b/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_communication_protocol.py similarity index 56% rename from src/utcp/client/transport_interfaces/mcp_transport.py rename to plugins/communication_protocols/mcp/src/utcp_mcp/mcp_communication_protocol.py index bc80912..6daf489 100644 --- a/src/utcp/client/transport_interfaces/mcp_transport.py +++ b/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_communication_protocol.py @@ -1,155 +1,170 @@ -import asyncio -import sys -from typing import Any, Dict, List, Optional, Callable -import logging +from typing import Any, Dict, Optional, AsyncGenerator, TYPE_CHECKING import json from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client from mcp.client.streamable_http import streamablehttp_client -from utcp.shared.provider import MCPProvider -from utcp.shared.tool import Tool -from utcp.shared.auth import OAuth2Auth +from utcp.data.utcp_manual import UtcpManual +from utcp.data.call_template import CallTemplate +from utcp.data.tool import Tool +from utcp.data.auth_implementations import OAuth2Auth +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.data.register_manual_response import RegisterManualResult import aiohttp from aiohttp import BasicAuth as AiohttpBasicAuth +from utcp_mcp.mcp_call_template import McpCallTemplate +if TYPE_CHECKING: + from utcp.utcp_client import UtcpClient +import logging +logger = logging.getLogger(__name__) -class MCPTransport: +class McpCommunicationProtocol(CommunicationProtocol): """MCP transport implementation that connects to MCP servers via stdio or HTTP. This implementation uses a session-per-operation approach where each operation (register, call_tool) opens a fresh session, performs the operation, and closes. """ - def __init__(self, logger: Optional[Callable[[str, Any], None]] = None): + def __init__(self): self._oauth_tokens: Dict[str, Dict[str, Any]] = {} - self._log = logger or (lambda *args, **kwargs: None) - def _log(self, message: str, error: bool = False): - """Log messages with appropriate level.""" - if error: - logging.error(f"[MCPTransport Error] {message}") - else: - logging.info(f"[MCPTransport Info] {message}") - - async def _list_tools_with_session(self, server_config, auth=None): - """List tools by creating a session.""" + async def _list_tools_with_session(self, server_config: Dict[str, Any], auth: Optional[OAuth2Auth] = None): # Create client streams based on transport type - if server_config.transport == "stdio": - params = StdioServerParameters( - command=server_config.command, - args=server_config.args, - env=server_config.env - ) + if "command" in server_config and "args" in server_config: + params = StdioServerParameters(**server_config) async with stdio_client(params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() tools_response = await session.list_tools() return tools_response.tools - elif server_config.transport == "http": + elif "url" in server_config: # Get authentication token if OAuth2 is configured auth_header = None if auth and isinstance(auth, OAuth2Auth): token = await self._handle_oauth2(auth) auth_header = {"Authorization": f"Bearer {token}"} - async with streamablehttp_client(server_config.url, auth=auth_header) as (read, write, _): + async with streamablehttp_client(server_config["url"], auth=auth_header) as (read, write, _): async with ClientSession(read, write) as session: await session.initialize() tools_response = await session.list_tools() return tools_response.tools else: - raise ValueError(f"Unsupported MCP transport: {server_config.transport}") + raise ValueError(f"Unsupported MCP transport: {json.dumps(server_config)}") - async def _call_tool_with_session(self, server_config, tool_name, inputs, auth=None): - """Call a tool by creating a session.""" - # Create client streams based on transport type - if server_config.transport == "stdio": - params = StdioServerParameters( - command=server_config.command, - args=server_config.args, - env=server_config.env - ) + async def _call_tool_with_session(self, server_config: Dict[str, Any], tool_name: str, inputs: Dict[str, Any], auth: Optional[OAuth2Auth] = None): + if "command" in server_config and "args" in server_config: + params = StdioServerParameters(**server_config) async with stdio_client(params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() result = await session.call_tool(tool_name, arguments=inputs) return result - elif server_config.transport == "http": + elif "url" in server_config: # Get authentication token if OAuth2 is configured auth_header = None if auth and isinstance(auth, OAuth2Auth): token = await self._handle_oauth2(auth) auth_header = {"Authorization": f"Bearer {token}"} - - async with streamablehttp_client(server_config.url, auth=auth_header) as (read, write, _): + + async with streamablehttp_client( + url=server_config["url"], + headers=server_config.get("headers", None), + timeout=server_config.get("timeout", 30), + sse_read_timeout=server_config.get("sse_read_timeout", 60 * 5), + terminate_on_close=server_config.get("terminate_on_close", True), + auth=auth_header + ) as (read, write, _): async with ClientSession(read, write) as session: await session.initialize() result = await session.call_tool(tool_name, arguments=inputs) return result else: - raise ValueError(f"Unsupported MCP transport: {server_config.transport}") + raise ValueError(f"Unsupported MCP transport: {json.dumps(server_config)}") - async def register_tool_provider(self, manual_provider: MCPProvider) -> List[Tool]: - """Register an MCP provider and discover its tools.""" + async def register_manual(self, caller: 'UtcpClient', manual_call_template: CallTemplate) -> RegisterManualResult: + if not isinstance(manual_call_template, McpCallTemplate): + raise ValueError("manual_call_template must be a McpCallTemplate") all_tools = [] - if manual_provider.config and manual_provider.config.mcpServers: - for server_name, server_config in manual_provider.config.mcpServers.items(): + errors = [] + if manual_call_template.config and manual_call_template.config.mcpServers: + for server_name, server_config in manual_call_template.config.mcpServers.items(): try: - self._log(f"Discovering tools for server '{server_name}' via {server_config.transport}") - tools = await self._list_tools_with_session(server_config, auth=manual_provider.auth) - self._log(f"Discovered {len(tools)} tools for server '{server_name}'") - all_tools.extend(tools) + logger.info(f"Discovering tools for server '{server_name}' via {server_config}") + mcp_tools = await self._list_tools_with_session(server_config, auth=manual_call_template.auth) + logger.info(f"Discovered {len(mcp_tools)} tools for server '{server_name}'") + for mcp_tool in mcp_tools: + # Convert mcp.Tool to utcp.data.tool.Tool + utcp_tool = Tool( + name=mcp_tool.name, + description=mcp_tool.description, + input_schema=mcp_tool.inputSchema, + output_schema=mcp_tool.outputSchema, + tool_call_template=manual_call_template + ) + all_tools.append(utcp_tool) except Exception as e: - self._log(f"Failed to discover tools for server '{server_name}': {e}", error=True) - return all_tools + logger.error(f"Failed to discover tools for server '{server_name}': {e}") + errors.append(f"Failed to discover tools for server '{server_name}': {e}") + return RegisterManualResult( + manual_call_template=manual_call_template, + manual=UtcpManual( + tools=all_tools + ), + success=len(errors) == 0, + errors=errors + ) - async def call_tool(self, tool_name: str, inputs: Dict[str, Any], tool_provider: MCPProvider) -> Any: - """Call a tool by creating a fresh session to the appropriate server.""" - if not tool_provider.config or not tool_provider.config.mcpServers: + async def call_tool(self, caller: 'UtcpClient', tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + if not isinstance(tool_call_template, McpCallTemplate): + raise ValueError("tool_call_template must be a McpCallTemplate") + if not tool_call_template.config or not tool_call_template.config.mcpServers: raise ValueError(f"No server configuration found for tool '{tool_name}'") # Try each server until we find one that has the tool - for server_name, server_config in tool_provider.config.mcpServers.items(): + for server_name, server_config in tool_call_template.config.mcpServers.items(): try: - self._log(f"Attempting to call tool '{tool_name}' on server '{server_name}'") + logger.info(f"Attempting to call tool '{tool_name}' on server '{server_name}'") # First check if this server has the tool - tools = await self._list_tools_with_session(server_config, auth=tool_provider.auth) + tools = await self._list_tools_with_session(server_config, auth=tool_call_template.auth) tool_names = [tool.name for tool in tools] if tool_name not in tool_names: - self._log(f"Tool '{tool_name}' not found in server '{server_name}'") + logger.info(f"Tool '{tool_name}' not found in server '{server_name}'") continue # Try next server # Call the tool - result = await self._call_tool_with_session(server_config, tool_name, inputs, auth=tool_provider.auth) + result = await self._call_tool_with_session(server_config, tool_name, tool_args, auth=tool_call_template.auth) # Process the result return self._process_tool_result(result, tool_name) except Exception as e: - self._log(f"Error calling tool '{tool_name}' on server '{server_name}': {e}", error=True) + logger.error(f"Error calling tool '{tool_name}' on server '{server_name}': {e}") continue # Try next server raise ValueError(f"Tool '{tool_name}' not found in any configured server") + async def call_tool_streaming(self, caller: 'UtcpClient', tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> AsyncGenerator[Any, None]: + yield self.call_tool(caller, tool_name, tool_args, tool_call_template) + def _process_tool_result(self, result, tool_name: str) -> Any: - """Process the tool result and return the appropriate format.""" - self._log(f"Processing tool result for '{tool_name}', type: {type(result)}") + logger.info(f"Processing tool result for '{tool_name}', type: {type(result)}") # Check for structured output first if hasattr(result, 'structured_output'): - self._log(f"Found structured_output: {result.structured_output}") + logger.info(f"Found structured_output: {result.structured_output}") return result.structured_output # Process content if available if hasattr(result, 'content'): content = result.content - self._log(f"Content type: {type(content)}") + logger.info(f"Content type: {type(content)}") # Handle list content if isinstance(content, list): - self._log(f"Content is a list with {len(content)} items") + logger.info(f"Content is a list with {len(content)} items") if not content: return [] @@ -210,9 +225,9 @@ def _parse_text_content(self, text: str) -> Any: # Return as string return text - async def deregister_tool_provider(self, manual_provider: MCPProvider) -> None: - """Deregister an MCP provider. This is a no-op in session-per-operation mode.""" - self._log(f"Deregistering provider '{manual_provider.name}' (no-op in session-per-operation mode)") + async def deregister_manual(self, caller: 'UtcpClient', manual_call_template: CallTemplate) -> None: + """Deregister an MCP manual. This is a no-op in session-per-operation mode.""" + logger.info(f"Deregistering manual '{manual_call_template.name}' (no-op in session-per-operation mode)") pass async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: @@ -226,7 +241,7 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: async with aiohttp.ClientSession() as session: # Method 1: Send credentials in the request body try: - self._log(f"Attempting OAuth2 token fetch for '{client_id}' with credentials in body.") + logger.info(f"Attempting OAuth2 token fetch for '{client_id}' with credentials in body.") body_data = { 'grant_type': 'client_credentials', 'client_id': client_id, @@ -239,11 +254,11 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: self._oauth_tokens[client_id] = token_response return token_response["access_token"] except aiohttp.ClientError as e: - self._log(f"OAuth2 with credentials in body failed: {e}. Trying Basic Auth header.") + logger.error(f"OAuth2 with credentials in body failed: {e}. Trying Basic Auth header.") # Method 2: Send credentials as Basic Auth header try: - self._log(f"Attempting OAuth2 token fetch for '{client_id}' with Basic Auth header.") + logger.info(f"Attempting OAuth2 token fetch for '{client_id}' with Basic Auth header.") header_auth = AiohttpBasicAuth(client_id, auth_details.client_secret) header_data = { 'grant_type': 'client_credentials', @@ -255,10 +270,5 @@ async def _handle_oauth2(self, auth_details: OAuth2Auth) -> str: self._oauth_tokens[client_id] = token_response return token_response["access_token"] except aiohttp.ClientError as e: - self._log(f"OAuth2 with Basic Auth header also failed: {e}", error=True) + logger.error(f"OAuth2 with Basic Auth header also failed: {e}") raise e - - async def close(self) -> None: - """Close the transport. This is a no-op in session-per-operation mode.""" - self._log("Closing MCP transport (no-op in session-per-operation mode)") - pass diff --git a/tests/client/transport_interfaces/mock_http_mcp_server.py b/plugins/communication_protocols/mcp/tests/mock_http_mcp_server.py similarity index 96% rename from tests/client/transport_interfaces/mock_http_mcp_server.py rename to plugins/communication_protocols/mcp/tests/mock_http_mcp_server.py index d5dd96b..d3ac833 100644 --- a/tests/client/transport_interfaces/mock_http_mcp_server.py +++ b/plugins/communication_protocols/mcp/tests/mock_http_mcp_server.py @@ -2,7 +2,7 @@ Mock HTTP MCP server for testing the MCP transport with HTTP transport. """ from mcp.server.fastmcp import FastMCP -from typing import TypedDict, List, Any +from typing import TypedDict, List # Create a stateless HTTP MCP server mcp = FastMCP(name="MockHttpServer", stateless_http=True) diff --git a/tests/client/transport_interfaces/mock_mcp_server.py b/plugins/communication_protocols/mcp/tests/mock_mcp_server.py similarity index 100% rename from tests/client/transport_interfaces/mock_mcp_server.py rename to plugins/communication_protocols/mcp/tests/mock_mcp_server.py diff --git a/tests/client/transport_interfaces/test_mcp_http_transport.py b/plugins/communication_protocols/mcp/tests/test_mcp_http_transport.py similarity index 55% rename from tests/client/transport_interfaces/test_mcp_http_transport.py rename to plugins/communication_protocols/mcp/tests/test_mcp_http_transport.py index 0d21e29..f8c626c 100644 --- a/tests/client/transport_interfaces/test_mcp_http_transport.py +++ b/plugins/communication_protocols/mcp/tests/test_mcp_http_transport.py @@ -11,8 +11,8 @@ import socket from typing import List, Optional, Tuple -from utcp.client.transport_interfaces.mcp_transport import MCPTransport -from utcp.shared.provider import MCPProvider, McpConfig, McpHttpServer +from utcp_mcp.mcp_call_template import McpCallTemplate, McpConfig +from utcp_mcp.mcp_communication_protocol import McpCommunicationProtocol HTTP_SERVER_NAME = "mock_http_server" HTTP_SERVER_PORT = 8000 @@ -66,44 +66,44 @@ async def http_server_process() -> subprocess.Popen: @pytest_asyncio.fixture -def http_mcp_provider() -> MCPProvider: - """Provides an MCPProvider configured to connect to the mock HTTP server.""" - server_config = McpHttpServer( - url=f"http://127.0.0.1:{HTTP_SERVER_PORT}/mcp", - transport="http" - ) - return MCPProvider( +def http_mcp_provider() -> McpCallTemplate: + """Provides an McpCallTemplate configured to connect to the mock HTTP server.""" + server_config = { + "url": f"http://127.0.0.1:{HTTP_SERVER_PORT}/mcp", + "transport": "http" + } + return McpCallTemplate( name="mock_http_provider", - provider_type="mcp", + call_template_type="mcp", config=McpConfig(mcpServers={HTTP_SERVER_NAME: server_config}) ) @pytest_asyncio.fixture -async def transport() -> MCPTransport: - """Provides a clean MCPTransport instance.""" - t = MCPTransport() +async def transport() -> McpCommunicationProtocol: + """Provides a clean McpCommunicationProtocol instance.""" + t = McpCommunicationProtocol() yield t - await t.close() @pytest.mark.asyncio -async def test_http_register_provider_discovers_tools( - transport: MCPTransport, - http_mcp_provider: MCPProvider, +async def test_http_register_manual_discovers_tools( + transport: McpCommunicationProtocol, + http_mcp_provider: McpCallTemplate, http_server_process: subprocess.Popen ): - """Test that registering an HTTP MCP provider discovers the correct tools.""" - tools = await transport.register_tool_provider(http_mcp_provider) - assert len(tools) == 4 - + """Test that registering an HTTP MCP manual discovers the correct tools.""" + register_result = await transport.register_manual(None, http_mcp_provider) + assert register_result.success + assert len(register_result.manual.tools) == 4 + # Find the echo tool - echo_tool = next((tool for tool in tools if tool.name == "echo"), None) + echo_tool = next((tool for tool in register_result.manual.tools if tool.name == "echo"), None) assert echo_tool is not None assert "echoes back its input" in echo_tool.description - + # Check for other tools - tool_names = [tool.name for tool in tools] + tool_names = [tool.name for tool in register_result.manual.tools] assert "greet" in tool_names assert "list_items" in tool_names assert "add_numbers" in tool_names @@ -111,96 +111,85 @@ async def test_http_register_provider_discovers_tools( @pytest.mark.asyncio async def test_http_structured_output( - transport: MCPTransport, - http_mcp_provider: MCPProvider, + transport: McpCommunicationProtocol, + http_mcp_provider: McpCallTemplate, http_server_process: subprocess.Popen ): """Test that HTTP MCP tools with structured output work correctly.""" # Register the provider - await transport.register_tool_provider(http_mcp_provider) + await transport.register_manual(None, http_mcp_provider) # Call the echo tool and verify the result - result = await transport.call_tool("echo", {"message": "http_test"}, http_mcp_provider) + result = await transport.call_tool(None, "echo", {"message": "http_test"}, http_mcp_provider) assert result == {"reply": "you said: http_test"} @pytest.mark.asyncio async def test_http_unstructured_output( - transport: MCPTransport, - http_mcp_provider: MCPProvider, + transport: McpCommunicationProtocol, + http_mcp_provider: McpCallTemplate, http_server_process: subprocess.Popen ): """Test that HTTP MCP tools with unstructured output types work correctly.""" # Register the provider - await transport.register_tool_provider(http_mcp_provider) + await transport.register_manual(None, http_mcp_provider) # Call the greet tool and verify the result - result = await transport.call_tool("greet", {"name": "Alice"}, http_mcp_provider) + result = await transport.call_tool(None, "greet", {"name": "Alice"}, http_mcp_provider) assert result == "Hello, Alice!" @pytest.mark.asyncio async def test_http_list_output( - transport: MCPTransport, - http_mcp_provider: MCPProvider, + transport: McpCommunicationProtocol, + http_mcp_provider: McpCallTemplate, http_server_process: subprocess.Popen ): """Test that HTTP MCP tools returning lists work correctly.""" # Register the provider - await transport.register_tool_provider(http_mcp_provider) + await transport.register_manual(None, http_mcp_provider) # Call the list_items tool and verify the result - result = await transport.call_tool("list_items", {"count": 3}, http_mcp_provider) + result = await transport.call_tool(None, "list_items", {"count": 3}, http_mcp_provider) - # The result might be wrapped in a "result" field or returned directly - if isinstance(result, dict) and "result" in result: - items = result["result"] - else: - items = result - - assert isinstance(items, list) - assert len(items) == 3 - assert items[0] == "item_0" - assert items[1] == "item_1" - assert items[2] == "item_2" + assert isinstance(result, list) + assert len(result) == 3 + assert result[0] == "item_0" + assert result[1] == "item_1" + assert result[2] == "item_2" @pytest.mark.asyncio async def test_http_numeric_output( - transport: MCPTransport, - http_mcp_provider: MCPProvider, + transport: McpCommunicationProtocol, + http_mcp_provider: McpCallTemplate, http_server_process: subprocess.Popen ): """Test that HTTP MCP tools returning numeric values work correctly.""" # Register the provider - await transport.register_tool_provider(http_mcp_provider) + await transport.register_manual(None, http_mcp_provider) # Call the add_numbers tool and verify the result - result = await transport.call_tool("add_numbers", {"a": 5, "b": 7}, http_mcp_provider) + result = await transport.call_tool(None, "add_numbers", {"a": 5, "b": 7}, http_mcp_provider) - # The result might be wrapped in a "result" field or returned directly - if isinstance(result, dict) and "result" in result: - value = result["result"] - else: - value = result - - assert value == 12 + assert result == 12 @pytest.mark.asyncio -async def test_http_deregister_provider( - transport: MCPTransport, - http_mcp_provider: MCPProvider, +async def test_http_deregister_manual( + transport: McpCommunicationProtocol, + http_mcp_provider: McpCallTemplate, http_server_process: subprocess.Popen ): - """Test that deregistering an HTTP MCP provider works (no-op in session-per-operation mode).""" - # Register a provider - tools = await transport.register_tool_provider(http_mcp_provider) - assert len(tools) == 4 - + """Test that deregistering an HTTP MCP manual works (no-op in session-per-operation mode).""" + # Register a manual + register_result = await transport.register_manual(None, http_mcp_provider) + assert register_result.success + assert len(register_result.manual.tools) == 4 + # Deregister it (this is a no-op in session-per-operation mode) - await transport.deregister_tool_provider(http_mcp_provider) - + await transport.deregister_manual(None, http_mcp_provider) + # Should still be able to call tools since we create fresh sessions - result = await transport.call_tool("echo", {"message": "test"}, http_mcp_provider) + result = await transport.call_tool(None, "echo", {"message": "test"}, http_mcp_provider) assert result == {"reply": "you said: test"} diff --git a/plugins/communication_protocols/mcp/tests/test_mcp_transport.py b/plugins/communication_protocols/mcp/tests/test_mcp_transport.py new file mode 100644 index 0000000..62e216e --- /dev/null +++ b/plugins/communication_protocols/mcp/tests/test_mcp_transport.py @@ -0,0 +1,120 @@ +import sys +import os +import pytest +import pytest_asyncio + +from utcp_mcp.mcp_communication_protocol import McpCommunicationProtocol +from utcp_mcp.mcp_call_template import McpCallTemplate, McpConfig + +SERVER_NAME = "mock_stdio_server" + + +@pytest_asyncio.fixture +def mcp_manual() -> McpCallTemplate: + """Provides an McpCallTemplate configured to run the mock stdio server.""" + server_path = os.path.join(os.path.dirname(__file__), "mock_mcp_server.py") + server_config = { + "command": sys.executable, + "args": [server_path], + } + return McpCallTemplate( + name="mock_mcp_manual", + call_template_type="mcp", + config=McpConfig(mcpServers={SERVER_NAME: server_config}) + ) + + +@pytest_asyncio.fixture +async def transport() -> McpCommunicationProtocol: + """Provides a clean McpCommunicationProtocol instance.""" + t = McpCommunicationProtocol() + yield t + + +@pytest.mark.asyncio +async def test_register_manual_discovers_tools(transport: McpCommunicationProtocol, mcp_manual: McpCallTemplate): + """Verify that registering a manual discovers the correct tools.""" + register_result = await transport.register_manual(None, mcp_manual) + assert register_result.success + assert len(register_result.manual.tools) == 4 + + # Find the echo tool + echo_tool = next((tool for tool in register_result.manual.tools if tool.name == "echo"), None) + assert echo_tool is not None + assert "echoes back its input" in echo_tool.description + + # Check for other tools + tool_names = [tool.name for tool in register_result.manual.tools] + assert "greet" in tool_names + assert "list_items" in tool_names + assert "add_numbers" in tool_names + + +@pytest.mark.asyncio +async def test_call_tool_succeeds(transport: McpCommunicationProtocol, mcp_manual: McpCallTemplate): + """Verify a successful tool call after registration.""" + await transport.register_manual(None, mcp_manual) + + result = await transport.call_tool(None, "echo", {"message": "test"}, mcp_manual) + + assert result == {"reply": "you said: test"} + + +@pytest.mark.asyncio +async def test_call_tool_works_without_register(transport: McpCommunicationProtocol, mcp_manual: McpCallTemplate): + """Verify that calling a tool works without prior registration in session-per-operation mode.""" + result = await transport.call_tool(None, "echo", {"message": "test"}, mcp_manual) + assert result == {"reply": "you said: test"} + + +@pytest.mark.asyncio +async def test_structured_output_tool(transport: McpCommunicationProtocol, mcp_manual: McpCallTemplate): + """Test that tools with structured output (TypedDict) work correctly.""" + await transport.register_manual(None, mcp_manual) + + result = await transport.call_tool(None, "echo", {"message": "test"}, mcp_manual) + assert result == {"reply": "you said: test"} + + +@pytest.mark.asyncio +async def test_unstructured_string_output(transport: McpCommunicationProtocol, mcp_manual: McpCallTemplate): + """Test that tools returning plain strings work correctly.""" + await transport.register_manual(None, mcp_manual) + + result = await transport.call_tool(None, "greet", {"name": "Alice"}, mcp_manual) + assert result == "Hello, Alice!" + + +@pytest.mark.asyncio +async def test_list_output(transport: McpCommunicationProtocol, mcp_manual: McpCallTemplate): + """Test that tools returning lists work correctly.""" + await transport.register_manual(None, mcp_manual) + + result = await transport.call_tool(None, "list_items", {"count": 3}, mcp_manual) + + assert isinstance(result, list) + assert len(result) == 3 + assert result == ["item_0", "item_1", "item_2"] + + +@pytest.mark.asyncio +async def test_numeric_output(transport: McpCommunicationProtocol, mcp_manual: McpCallTemplate): + """Test that tools returning numeric values work correctly.""" + await transport.register_manual(None, mcp_manual) + + result = await transport.call_tool(None, "add_numbers", {"a": 5, "b": 7}, mcp_manual) + + assert result == 12 + + +@pytest.mark.asyncio +async def test_deregister_manual(transport: McpCommunicationProtocol, mcp_manual: McpCallTemplate): + """Verify that deregistering a manual works (no-op in session-per-operation mode).""" + register_result = await transport.register_manual(None, mcp_manual) + assert register_result.success + assert len(register_result.manual.tools) == 4 + + await transport.deregister_manual(None, mcp_manual) + + result = await transport.call_tool(None, "echo", {"message": "test"}, mcp_manual) + assert result == {"reply": "you said: test"} diff --git a/plugins/communication_protocols/socket/INCOMPLETE b/plugins/communication_protocols/socket/INCOMPLETE new file mode 100644 index 0000000..e69de29 diff --git a/plugins/communication_protocols/socket/old_tests/__init__.py b/plugins/communication_protocols/socket/old_tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/plugins/communication_protocols/socket/old_tests/test_tcp_transport.py b/plugins/communication_protocols/socket/old_tests/test_tcp_transport.py new file mode 100644 index 0000000..3e4b33a --- /dev/null +++ b/plugins/communication_protocols/socket/old_tests/test_tcp_transport.py @@ -0,0 +1,875 @@ +# import pytest +# import pytest_asyncio +# import json +# import asyncio +# import socket +# import struct +# import threading +# from unittest.mock import MagicMock, patch, AsyncMock + +# from utcp.client.transport_interfaces.tcp_transport import TCPTransport +# from utcp.shared.provider import TCPProvider +# from utcp.shared.tool import Tool, ToolInputOutputSchema + + +# class MockTCPServer: +# """Mock TCP server for testing.""" + +# def __init__(self, host='localhost', port=0, response_delay=0.0): +# self.host = host +# self.port = port +# self.sock = None +# self.running = False +# self.responses = {} # Map message -> response +# self.call_count = 0 +# self.server_task = None +# self.connections = [] +# self.response_delay = response_delay # Delay before sending response (seconds) + +# async def start(self): +# """Start the mock TCP server.""" +# # Create socket and bind +# self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) +# self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) +# self.sock.bind((self.host, self.port)) +# if self.port == 0: # Auto-assign port +# self.port = self.sock.getsockname()[1] + +# self.sock.listen(5) +# self.running = True + +# # Start listening task +# self.server_task = asyncio.create_task(self._accept_connections()) + +# # Give the server a moment to start +# await asyncio.sleep(0.1) + +# async def stop(self): +# """Stop the mock TCP server.""" +# self.running = False +# if self.server_task: +# self.server_task.cancel() +# try: +# await self.server_task +# except asyncio.CancelledError: +# pass + +# # Close all active connections +# for conn in self.connections: +# try: +# conn.close() +# except Exception: +# pass +# self.connections.clear() + +# if self.sock: +# self.sock.close() + +# async def _accept_connections(self): +# """Accept incoming TCP connections.""" +# self.sock.setblocking(False) + +# while self.running: +# try: +# conn, addr = await asyncio.get_event_loop().sock_accept(self.sock) +# self.connections.append(conn) +# # Handle each connection in a separate task +# asyncio.create_task(self._handle_connection(conn, addr)) +# except asyncio.CancelledError: +# break +# except Exception as e: +# if self.running: +# print(f"Mock TCP server accept error: {e}") +# await asyncio.sleep(0.01) + +# async def _handle_connection(self, conn, addr): +# """Handle a single TCP connection.""" +# try: +# # Read data from client +# data = await asyncio.get_event_loop().sock_recv(conn, 4096) +# if not data: +# return + +# self.call_count += 1 + +# try: +# message = data.decode('utf-8') +# except UnicodeDecodeError: +# message = data.hex() # Fallback for binary data + +# # Get response for this message +# response = self.responses.get(message, '{"error": "unknown_message"}') + +# # Convert response to bytes +# if isinstance(response, str): +# response_bytes = response.encode('utf-8') +# elif isinstance(response, bytes): +# response_bytes = response +# elif isinstance(response, dict) or isinstance(response, list): +# response_bytes = json.dumps(response).encode('utf-8') +# else: +# response_bytes = str(response).encode('utf-8') + +# # Add delay if configured +# if self.response_delay > 0: +# await asyncio.sleep(self.response_delay) + +# # Send response back +# await asyncio.get_event_loop().sock_sendall(conn, response_bytes) + +# except Exception as e: +# if self.running: +# print(f"Mock TCP server connection error: {e}") +# finally: +# try: +# conn.close() +# except Exception: +# pass +# if conn in self.connections: +# self.connections.remove(conn) + +# def set_response(self, message, response): +# """Set a response for a specific message.""" +# self.responses[message] = response + + +# class MockTCPServerWithFraming(MockTCPServer): +# """Mock TCP server that handles different framing strategies.""" + +# def __init__(self, host='localhost', port=0, framing_strategy='stream', response_delay=0.0): +# super().__init__(host, port, response_delay) +# self.framing_strategy = framing_strategy +# self.length_prefix_bytes = 4 +# self.length_prefix_endian = 'big' +# self.message_delimiter = '\n' +# self.fixed_message_length = None + +# async def _handle_connection(self, conn, addr): +# """Handle a single TCP connection with framing.""" +# try: +# if self.framing_strategy == 'length_prefix': +# # Read length prefix first +# length_data = await asyncio.get_event_loop().sock_recv(conn, self.length_prefix_bytes) +# if not length_data: +# return + +# if self.length_prefix_bytes == 1: +# message_length = struct.unpack(f"{'>' if self.length_prefix_endian == 'big' else '<'}B", length_data)[0] +# elif self.length_prefix_bytes == 2: +# message_length = struct.unpack(f"{'>' if self.length_prefix_endian == 'big' else '<'}H", length_data)[0] +# elif self.length_prefix_bytes == 4: +# message_length = struct.unpack(f"{'>' if self.length_prefix_endian == 'big' else '<'}I", length_data)[0] + +# # Read the actual message +# data = await asyncio.get_event_loop().sock_recv(conn, message_length) + +# elif self.framing_strategy == 'delimiter': +# # Read until delimiter +# data = b'' +# delimiter_bytes = self.message_delimiter.encode('utf-8') +# while not data.endswith(delimiter_bytes): +# chunk = await asyncio.get_event_loop().sock_recv(conn, 1) +# if not chunk: +# break +# data += chunk +# # Remove delimiter +# data = data[:-len(delimiter_bytes)] + +# elif self.framing_strategy == 'fixed_length': +# # Read fixed number of bytes +# data = await asyncio.get_event_loop().sock_recv(conn, self.fixed_message_length) + +# else: # stream +# # Read all available data +# data = await asyncio.get_event_loop().sock_recv(conn, 4096) + +# if not data: +# return + +# self.call_count += 1 + +# try: +# message = data.decode('utf-8') +# except UnicodeDecodeError: +# message = data.hex() + +# # Get response for this message +# response = self.responses.get(message, '{"error": "unknown_message"}') + +# # Convert response to bytes +# if isinstance(response, str): +# response_bytes = response.encode('utf-8') +# elif isinstance(response, bytes): +# response_bytes = response +# elif isinstance(response, dict) or isinstance(response, list): +# response_bytes = json.dumps(response).encode('utf-8') +# else: +# response_bytes = str(response).encode('utf-8') + +# # Add delay if configured +# if self.response_delay > 0: +# await asyncio.sleep(self.response_delay) + +# # Send response with appropriate framing +# if self.framing_strategy == 'length_prefix': +# # Add length prefix +# length = len(response_bytes) +# if self.length_prefix_bytes == 1: +# length_bytes = struct.pack(f"{'>' if self.length_prefix_endian == 'big' else '<'}B", length) +# elif self.length_prefix_bytes == 2: +# length_bytes = struct.pack(f"{'>' if self.length_prefix_endian == 'big' else '<'}H", length) +# elif self.length_prefix_bytes == 4: +# length_bytes = struct.pack(f"{'>' if self.length_prefix_endian == 'big' else '<'}I", length) + +# await asyncio.get_event_loop().sock_sendall(conn, length_bytes + response_bytes) + +# elif self.framing_strategy == 'delimiter': +# # Add delimiter +# delimiter_bytes = self.message_delimiter.encode('utf-8') +# await asyncio.get_event_loop().sock_sendall(conn, response_bytes + delimiter_bytes) + +# else: # stream or fixed_length +# await asyncio.get_event_loop().sock_sendall(conn, response_bytes) + +# except Exception as e: +# if self.running: +# print(f"Mock TCP server connection error: {e}") +# finally: +# try: +# conn.close() +# except Exception: +# pass +# if conn in self.connections: +# self.connections.remove(conn) + + +# @pytest_asyncio.fixture +# async def mock_tcp_server(): +# """Create a mock TCP server for testing.""" +# server = MockTCPServer() +# await server.start() +# yield server +# await server.stop() + + +# @pytest_asyncio.fixture +# async def mock_tcp_server_length_prefix(): +# """Create a mock TCP server with length-prefix framing.""" +# server = MockTCPServerWithFraming(framing_strategy='length_prefix') +# await server.start() +# yield server +# await server.stop() + + +# @pytest_asyncio.fixture +# async def mock_tcp_server_delimiter(): +# """Create a mock TCP server with delimiter framing.""" +# server = MockTCPServerWithFraming(framing_strategy='delimiter') +# await server.start() +# yield server +# await server.stop() + + +# @pytest_asyncio.fixture +# async def mock_tcp_server_slow(): +# """Create a mock TCP server with a 2-second response delay.""" +# server = MockTCPServer(response_delay=2.0) # 2-second delay +# await server.start() +# yield server +# await server.stop() + + +# @pytest.fixture +# def logger(): +# """Create a mock logger.""" +# return MagicMock() + + +# @pytest.fixture +# def tcp_transport(logger): +# """Create a TCP transport instance.""" +# return TCPTransport(logger=logger) + + +# @pytest.fixture +# def tcp_provider(mock_tcp_server): +# """Create a basic TCP provider for testing.""" +# return TCPProvider( +# name="test_tcp_provider", +# host=mock_tcp_server.host, +# port=mock_tcp_server.port, +# request_data_format="json", +# response_byte_format="utf-8", +# framing_strategy="stream", +# timeout=5000 +# ) + + +# @pytest.fixture +# def text_template_provider(mock_tcp_server): +# """Create a TCP provider with text template format.""" +# return TCPProvider( +# name="text_template_provider", +# host=mock_tcp_server.host, +# port=mock_tcp_server.port, +# request_data_format="text", +# request_data_template="ACTION UTCP_ARG_cmd_UTCP_ARG PARAM UTCP_ARG_value_UTCP_ARG", +# response_byte_format="utf-8", +# framing_strategy="stream", +# timeout=5000 +# ) + + +# @pytest.fixture +# def raw_bytes_provider(mock_tcp_server): +# """Create a TCP provider that returns raw bytes.""" +# return TCPProvider( +# name="raw_bytes_provider", +# host=mock_tcp_server.host, +# port=mock_tcp_server.port, +# request_data_format="json", +# response_byte_format=None, # Raw bytes +# framing_strategy="stream", +# timeout=5000 +# ) + + +# @pytest.fixture +# def length_prefix_provider(mock_tcp_server_length_prefix): +# """Create a TCP provider with length-prefix framing.""" +# return TCPProvider( +# name="length_prefix_provider", +# host=mock_tcp_server_length_prefix.host, +# port=mock_tcp_server_length_prefix.port, +# request_data_format="json", +# response_byte_format="utf-8", +# framing_strategy="length_prefix", +# length_prefix_bytes=4, +# length_prefix_endian="big", +# timeout=5000 +# ) + + +# @pytest.fixture +# def delimiter_provider(mock_tcp_server_delimiter): +# """Create a TCP provider with delimiter framing.""" +# return TCPProvider( +# name="delimiter_provider", +# host=mock_tcp_server_delimiter.host, +# port=mock_tcp_server_delimiter.port, +# request_data_format="json", +# response_byte_format="utf-8", +# framing_strategy="delimiter", +# message_delimiter="\n", +# timeout=5000 +# ) + + +# # Test register_tool_provider +# @pytest.mark.asyncio +# async def test_register_tool_provider(tcp_transport, tcp_provider, mock_tcp_server, logger): +# """Test registering a tool provider.""" +# # Set up discovery response +# discovery_response = { +# "tools": [ +# { +# "name": "test_tool", +# "description": "A test tool", +# "inputs": { +# "type": "object", +# "properties": { +# "param1": {"type": "string", "description": "First parameter"} +# }, +# "required": ["param1"] +# }, +# "outputs": { +# "type": "object", +# "properties": { +# "result": {"type": "string", "description": "Result"} +# } +# }, +# "tool_provider": tcp_provider.model_dump() +# } +# ] +# } + +# mock_tcp_server.set_response('{"type": "utcp"}', discovery_response) + +# # Register the provider +# tools = await tcp_transport.register_tool_provider(tcp_provider) + +# # Check results +# assert len(tools) == 1 +# assert tools[0].name == "test_tool" +# assert tools[0].description == "A test tool" +# assert mock_tcp_server.call_count == 1 + +# # Verify logger was called +# logger.assert_called() + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_empty_response(tcp_transport, tcp_provider, mock_tcp_server): +# """Test registering a tool provider with empty response.""" +# mock_tcp_server.set_response('{"type": "utcp"}', {"tools": []}) + +# tools = await tcp_transport.register_tool_provider(tcp_provider) + +# assert len(tools) == 0 +# assert mock_tcp_server.call_count == 1 + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_invalid_json(tcp_transport, tcp_provider, mock_tcp_server): +# """Test registering a tool provider with invalid JSON response.""" +# mock_tcp_server.set_response('{"type": "utcp"}', "invalid json response") + +# tools = await tcp_transport.register_tool_provider(tcp_provider) + +# assert len(tools) == 0 + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_invalid_provider_type(tcp_transport): +# """Test registering a non-TCP provider raises ValueError.""" +# from utcp.shared.provider import HttpProvider + +# invalid_provider = HttpProvider(url="http://example.com") + +# with pytest.raises(ValueError, match="TCPTransport can only be used with TCPProvider"): +# await tcp_transport.register_tool_provider(invalid_provider) + + +# # Test deregister_tool_provider +# @pytest.mark.asyncio +# async def test_deregister_tool_provider(tcp_transport, tcp_provider): +# """Test deregistering a tool provider (should be a no-op).""" +# # Should not raise any exceptions +# await tcp_transport.deregister_tool_provider(tcp_provider) + + +# @pytest.mark.asyncio +# async def test_deregister_tool_provider_invalid_type(tcp_transport): +# """Test deregistering a non-TCP provider raises ValueError.""" +# from utcp.shared.provider import HttpProvider + +# invalid_provider = HttpProvider(url="http://example.com") + +# with pytest.raises(ValueError, match="TCPTransport can only be used with TCPProvider"): +# await tcp_transport.deregister_tool_provider(invalid_provider) + + +# # Test call_tool with JSON format +# @pytest.mark.asyncio +# async def test_call_tool_json_format(tcp_transport, tcp_provider, mock_tcp_server): +# """Test calling a tool with JSON format.""" +# mock_tcp_server.set_response('{"param1": "value1"}', '{"result": "success"}') + +# arguments = {"param1": "value1"} +# result = await tcp_transport.call_tool("test_tool", arguments, tcp_provider) + +# assert result == '{"result": "success"}' +# assert mock_tcp_server.call_count == 1 + + +# @pytest.mark.asyncio +# async def test_call_tool_text_template_format(tcp_transport, text_template_provider, mock_tcp_server): +# """Test calling a tool with text template format.""" +# mock_tcp_server.set_response("ACTION get PARAM data123", '{"result": "template_success"}') + +# arguments = {"cmd": "get", "value": "data123"} +# result = await tcp_transport.call_tool("test_tool", arguments, text_template_provider) + +# assert result == '{"result": "template_success"}' +# assert mock_tcp_server.call_count == 1 + + +# @pytest.mark.asyncio +# async def test_call_tool_text_format_no_template(tcp_transport, mock_tcp_server): +# """Test calling a tool with text format but no template.""" +# provider = TCPProvider( +# name="no_template_provider", +# host=mock_tcp_server.host, +# port=mock_tcp_server.port, +# request_data_format="text", +# request_data_template=None, +# response_byte_format="utf-8", +# framing_strategy="stream", +# timeout=5000 +# ) + +# # Should use fallback format (space-separated values) +# mock_tcp_server.set_response("value1 value2", '{"result": "fallback_success"}') + +# arguments = {"param1": "value1", "param2": "value2"} +# result = await tcp_transport.call_tool("test_tool", arguments, provider) + +# assert result == '{"result": "fallback_success"}' + + +# @pytest.mark.asyncio +# async def test_call_tool_raw_bytes_response(tcp_transport, raw_bytes_provider, mock_tcp_server): +# """Test calling a tool that returns raw bytes.""" +# binary_response = b'\x01\x02\x03\x04' +# mock_tcp_server.set_response('{"param1": "value1"}', binary_response) + +# arguments = {"param1": "value1"} +# result = await tcp_transport.call_tool("test_tool", arguments, raw_bytes_provider) + +# assert result == binary_response +# assert isinstance(result, bytes) + + +# @pytest.mark.asyncio +# async def test_call_tool_invalid_provider_type(tcp_transport): +# """Test calling a tool with non-TCP provider raises ValueError.""" +# from utcp.shared.provider import HttpProvider + +# invalid_provider = HttpProvider(url="http://example.com") + +# with pytest.raises(ValueError, match="TCPTransport can only be used with TCPProvider"): +# await tcp_transport.call_tool("test_tool", {}, invalid_provider) + + +# # Test framing strategies +# @pytest.mark.asyncio +# async def test_call_tool_length_prefix_framing(tcp_transport, length_prefix_provider, mock_tcp_server_length_prefix): +# """Test calling a tool with length-prefix framing.""" +# mock_tcp_server_length_prefix.set_response('{"param1": "value1"}', '{"result": "length_prefix_success"}') + +# arguments = {"param1": "value1"} +# result = await tcp_transport.call_tool("test_tool", arguments, length_prefix_provider) + +# assert result == '{"result": "length_prefix_success"}' + + +# @pytest.mark.asyncio +# async def test_call_tool_delimiter_framing(tcp_transport, delimiter_provider, mock_tcp_server_delimiter): +# """Test calling a tool with delimiter framing.""" +# mock_tcp_server_delimiter.set_response('{"param1": "value1"}', '{"result": "delimiter_success"}') + +# arguments = {"param1": "value1"} +# result = await tcp_transport.call_tool("test_tool", arguments, delimiter_provider) + +# assert result == '{"result": "delimiter_success"}' + + +# @pytest.mark.asyncio +# async def test_call_tool_fixed_length_framing(tcp_transport, mock_tcp_server): +# """Test calling a tool with fixed-length framing.""" +# provider = TCPProvider( +# name="fixed_length_provider", +# host=mock_tcp_server.host, +# port=mock_tcp_server.port, +# request_data_format="json", +# response_byte_format="utf-8", +# framing_strategy="fixed_length", +# fixed_message_length=20, +# timeout=5000 +# ) + +# # Set up server to handle fixed-length messages +# mock_tcp_server.responses['{"param1": "value1"}'] = '{"result": "fixed"}'.ljust(20) # Pad to 20 bytes + +# arguments = {"param1": "value1"} +# result = await tcp_transport.call_tool("test_tool", arguments, provider) + +# assert '{"result": "fixed"}' in result + + +# # Test message formatting +# def test_format_tool_call_message_json(tcp_transport): +# """Test formatting tool call message with JSON format.""" +# provider = TCPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="json" +# ) + +# arguments = {"param1": "value1", "param2": 123} +# result = tcp_transport._format_tool_call_message(arguments, provider) + +# assert result == json.dumps(arguments) + + +# def test_format_tool_call_message_text_with_template(tcp_transport): +# """Test formatting tool call message with text template.""" +# provider = TCPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="text", +# request_data_template="ACTION UTCP_ARG_cmd_UTCP_ARG PARAM UTCP_ARG_value_UTCP_ARG" +# ) + +# arguments = {"cmd": "get", "value": "data123"} +# result = tcp_transport._format_tool_call_message(arguments, provider) + +# # Should substitute placeholders +# assert result == "ACTION get PARAM data123" + + +# def test_format_tool_call_message_text_with_complex_values(tcp_transport): +# """Test formatting tool call message with complex values in template.""" +# provider = TCPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="text", +# request_data_template="DATA UTCP_ARG_obj_UTCP_ARG" +# ) + +# arguments = {"obj": {"nested": "value", "number": 123}} +# result = tcp_transport._format_tool_call_message(arguments, provider) + +# # Should JSON-serialize complex values +# assert result == 'DATA {"nested": "value", "number": 123}' + + +# def test_format_tool_call_message_text_no_template(tcp_transport): +# """Test formatting tool call message with text format but no template.""" +# provider = TCPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="text", +# request_data_template=None +# ) + +# arguments = {"param1": "value1", "param2": "value2"} +# result = tcp_transport._format_tool_call_message(arguments, provider) + +# # Should use fallback format (space-separated values) +# assert result == "value1 value2" + + +# def test_format_tool_call_message_default_to_json(tcp_transport): +# """Test formatting tool call message defaults to JSON for unknown format.""" +# # Create a provider with valid format first +# provider = TCPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="json" +# ) + +# # Manually set an invalid format to test the fallback behavior +# provider.request_data_format = "unknown" # Invalid format + +# arguments = {"param1": "value1"} +# result = tcp_transport._format_tool_call_message(arguments, provider) + +# # Should default to JSON +# assert result == json.dumps(arguments) + + +# # Test framing encoding and decoding +# def test_encode_message_with_length_prefix_framing(tcp_transport): +# """Test encoding message with length-prefix framing.""" +# provider = TCPProvider( +# name="test", +# host="localhost", +# port=1234, +# framing_strategy="length_prefix", +# length_prefix_bytes=4, +# length_prefix_endian="big" +# ) + +# message = "test message" +# result = tcp_transport._encode_message_with_framing(message, provider) + +# # Should have 4-byte big-endian length prefix +# expected_length = len(message.encode('utf-8')) +# expected_prefix = struct.pack('>I', expected_length) + +# assert result.startswith(expected_prefix) +# assert result[4:] == message.encode('utf-8') + + +# def test_encode_message_with_delimiter_framing(tcp_transport): +# """Test encoding message with delimiter framing.""" +# provider = TCPProvider( +# name="test", +# host="localhost", +# port=1234, +# framing_strategy="delimiter", +# message_delimiter="\n" +# ) + +# message = "test message" +# result = tcp_transport._encode_message_with_framing(message, provider) + +# # Should have delimiter appended +# assert result == (message + "\n").encode('utf-8') + + +# def test_encode_message_with_stream_framing(tcp_transport): +# """Test encoding message with stream framing.""" +# provider = TCPProvider( +# name="test", +# host="localhost", +# port=1234, +# framing_strategy="stream" +# ) + +# message = "test message" +# result = tcp_transport._encode_message_with_framing(message, provider) + +# # Should just be the raw message +# assert result == message.encode('utf-8') + + +# # Test error handling and edge cases +# @pytest.mark.asyncio +# async def test_call_tool_server_error(tcp_transport, tcp_provider, mock_tcp_server): +# """Test handling server errors during tool calls.""" +# # Don't set any response, so the server will return an error +# arguments = {"param1": "value1"} + +# # Call the tool - should get the default error response +# result = await tcp_transport.call_tool("test_tool", arguments, tcp_provider) + +# # Should receive the default error message +# assert '{"error": "unknown_message"}' in result + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_malformed_tool(tcp_transport, tcp_provider, mock_tcp_server): +# """Test registering provider with malformed tool definition.""" +# # Set up discovery response with invalid tool +# discovery_response = { +# "tools": [ +# { +# "name": "test_tool", +# # Missing required fields like inputs, outputs, tool_provider +# } +# ] +# } + +# mock_tcp_server.set_response('{"type": "utcp"}', discovery_response) + +# # Register the provider - should handle invalid tool gracefully +# tools = await tcp_transport.register_tool_provider(tcp_provider) + +# # Should return empty list due to invalid tool definition +# assert len(tools) == 0 + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_bytes_response(tcp_transport, tcp_provider, mock_tcp_server): +# """Test registering provider that returns bytes response.""" +# # Set up discovery response as JSON but provider returns raw bytes +# discovery_response = '{"tools": []}'.encode('utf-8') + +# mock_tcp_server.set_response('{"type": "utcp"}', discovery_response) + +# # Register the provider - should handle bytes response by decoding +# tools = await tcp_transport.register_tool_provider(tcp_provider) + +# # Should successfully decode and parse +# assert len(tools) == 0 + + +# # Test logging functionality +# @pytest.mark.asyncio +# async def test_logging_calls(tcp_transport, tcp_provider, mock_tcp_server, logger): +# """Test that logging functions are called appropriately.""" +# # Set up discovery response +# discovery_response = {"tools": []} +# mock_tcp_server.set_response('{"type": "utcp"}', discovery_response) + +# # Register provider +# await tcp_transport.register_tool_provider(tcp_provider) + +# # Verify logger was called +# logger.assert_called() + +# # Call tool +# mock_tcp_server.set_response('{}', {"result": "test"}) +# await tcp_transport.call_tool("test_tool", {}, tcp_provider) + +# # Logger should have been called multiple times +# assert logger.call_count > 1 + + +# # Test timeout handling +# @pytest.mark.asyncio +# async def test_call_tool_timeout(tcp_transport): +# """Test calling a tool with timeout using delimiter framing.""" +# # Create a slow server with delimiter framing +# slow_server = MockTCPServerWithFraming( +# framing_strategy='delimiter', +# response_delay=2.0 # 2-second delay +# ) +# await slow_server.start() + +# try: +# # Create provider with 1-second timeout, but server has 2-second delay +# provider = TCPProvider( +# name="timeout_provider", +# host=slow_server.host, +# port=slow_server.port, +# request_data_format="json", +# response_byte_format="utf-8", +# framing_strategy="delimiter", +# message_delimiter="\n", +# timeout=1000 # 1 second timeout, but server delays 2 seconds +# ) + +# # Set up a response (server will delay 2 seconds before responding) +# slow_server.set_response('{"param1": "value1"}', '{"result": "delayed_response"}') + +# arguments = {"param1": "value1"} + +# # Should timeout because server takes 2 seconds but timeout is 1 second +# # Delimiter framing will treat timeout as an error since it expects a complete message +# with pytest.raises(Exception): # Expect timeout error +# await tcp_transport.call_tool("test_tool", arguments, provider) +# finally: +# await slow_server.stop() + + +# @pytest.mark.asyncio +# async def test_call_tool_connection_refused(tcp_transport): +# """Test calling a tool when connection is refused.""" +# # Use a port that's definitely not listening +# provider = TCPProvider( +# name="refused_provider", +# host="localhost", +# port=1, # Port 1 should be refused +# request_data_format="json", +# response_byte_format="utf-8", +# framing_strategy="stream", +# timeout=5000 +# ) + +# arguments = {"param1": "value1"} + +# # Should handle connection error gracefully +# with pytest.raises(Exception): # Expect connection refused or similar +# await tcp_transport.call_tool("test_tool", arguments, provider) + + +# # Test different byte encodings +# @pytest.mark.asyncio +# async def test_call_tool_different_encodings(tcp_transport, mock_tcp_server): +# """Test calling a tool with different response byte encodings.""" +# # Test ASCII encoding +# provider_ascii = TCPProvider( +# name="ascii_provider", +# host=mock_tcp_server.host, +# port=mock_tcp_server.port, +# request_data_format="json", +# response_byte_format="ascii", +# framing_strategy="stream", +# timeout=5000 +# ) + +# mock_tcp_server.set_response('{"param1": "value1"}', '{"result": "ascii_success"}') + +# arguments = {"param1": "value1"} +# result = await tcp_transport.call_tool("test_tool", arguments, provider_ascii) + +# assert result == '{"result": "ascii_success"}' +# assert isinstance(result, str) diff --git a/plugins/communication_protocols/socket/old_tests/test_udp_transport.py b/plugins/communication_protocols/socket/old_tests/test_udp_transport.py new file mode 100644 index 0000000..2ba396e --- /dev/null +++ b/plugins/communication_protocols/socket/old_tests/test_udp_transport.py @@ -0,0 +1,625 @@ +# import pytest +# import pytest_asyncio +# import json +# import asyncio +# import socket +# from unittest.mock import MagicMock, patch, AsyncMock + +# from utcp.client.transport_interfaces.udp_transport import UDPTransport +# from utcp.shared.provider import UDPProvider +# from utcp.shared.tool import Tool, ToolInputOutputSchema + + +# class MockUDPServer: +# """Mock UDP server for testing.""" + +# def __init__(self, host='localhost', port=0): +# self.host = host +# self.port = port +# self.sock = None +# self.running = False +# self.responses = {} # Map message -> response +# self.call_count = 0 +# self.listen_task = None + +# async def start(self): +# """Start the mock UDP server.""" +# # Create socket and bind +# self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) +# # Keep it blocking since we're using run_in_executor +# self.sock.bind((self.host, self.port)) +# if self.port == 0: # Auto-assign port +# self.port = self.sock.getsockname()[1] + +# self.running = True + +# # Start listening task +# self.listen_task = asyncio.create_task(self._listen()) + +# # Give the server a moment to start +# await asyncio.sleep(0.1) + +# async def stop(self): +# """Stop the mock UDP server.""" +# self.running = False +# if self.listen_task: +# self.listen_task.cancel() +# try: +# await self.listen_task +# except asyncio.CancelledError: +# pass +# if self.sock: +# self.sock.close() + +# async def _listen(self): +# """Listen for UDP messages and send responses.""" +# # Use a blocking approach with short timeout for responsiveness +# self.sock.settimeout(0.01) # Very short timeout + +# while self.running: +# try: +# data, addr = self.sock.recvfrom(4096) +# self.call_count += 1 + +# try: +# message = data.decode('utf-8') +# except UnicodeDecodeError: +# message = data.hex() # Fallback for binary data + +# # Get response for this message +# response = self.responses.get(message, '{"error": "unknown_message"}') + +# # Convert response to bytes +# if isinstance(response, str): +# response_bytes = response.encode('utf-8') +# elif isinstance(response, bytes): +# response_bytes = response +# elif isinstance(response, dict) or isinstance(response, list): +# response_bytes = json.dumps(response).encode('utf-8') +# else: +# response_bytes = str(response).encode('utf-8') + +# # Send response back immediately +# self.sock.sendto(response_bytes, addr) + +# except socket.timeout: +# # Expected timeout, continue loop +# await asyncio.sleep(0.001) # Brief async yield +# continue +# except asyncio.CancelledError: +# break +# except Exception as e: +# if self.running: # Only log if we're still supposed to be running +# import traceback +# print(f"Mock UDP server error: {e}") +# print(f"Traceback: {traceback.format_exc()}") +# await asyncio.sleep(0.01) # Brief pause before retrying + +# def set_response(self, message, response): +# """Set a response for a specific message.""" +# self.responses[message] = response + + +# @pytest_asyncio.fixture +# async def mock_udp_server(): +# """Create a mock UDP server for testing.""" +# server = MockUDPServer() +# await server.start() +# yield server +# await server.stop() + + +# @pytest.fixture +# def logger(): +# """Create a mock logger.""" +# return MagicMock() + + +# @pytest.fixture +# def udp_transport(logger): +# """Create a UDP transport instance.""" +# return UDPTransport(logger=logger) + + +# @pytest.fixture +# def udp_provider(mock_udp_server): +# """Create a basic UDP provider for testing.""" +# return UDPProvider( +# name="test_udp_provider", +# host=mock_udp_server.host, +# port=mock_udp_server.port, +# number_of_response_datagrams=1, +# request_data_format="json", +# response_byte_format="utf-8", +# timeout=5000 +# ) + + +# @pytest.fixture +# def text_template_provider(mock_udp_server): +# """Create a UDP provider with text template format.""" +# return UDPProvider( +# name="test_text_template_provider", +# host=mock_udp_server.host, +# port=mock_udp_server.port, +# number_of_response_datagrams=1, +# request_data_format="text", +# request_data_template="COMMAND UTCP_ARG_action_UTCP_ARG UTCP_ARG_value_UTCP_ARG", +# response_byte_format="utf-8", +# timeout=5000 +# ) + + +# @pytest.fixture +# def raw_bytes_provider(mock_udp_server): +# """Create a UDP provider that returns raw bytes.""" +# return UDPProvider( +# name="test_raw_bytes_provider", +# host=mock_udp_server.host, +# port=mock_udp_server.port, +# number_of_response_datagrams=1, +# request_data_format="json", +# response_byte_format=None, # Return raw bytes +# timeout=5000 +# ) + + +# @pytest.fixture +# def multi_datagram_provider(mock_udp_server): +# """Create a UDP provider that expects multiple response datagrams.""" +# return UDPProvider( +# name="test_multi_datagram_provider", +# host=mock_udp_server.host, +# port=mock_udp_server.port, +# number_of_response_datagrams=3, +# request_data_format="json", +# response_byte_format="utf-8", +# timeout=5000 +# ) + + +# # Test register_tool_provider +# @pytest.mark.asyncio +# async def test_register_tool_provider(udp_transport, udp_provider, mock_udp_server, logger): +# """Test registering a tool provider.""" +# # Set up discovery response +# discovery_response = { +# "tools": [ +# { +# "name": "test_tool", +# "description": "Test tool", +# "inputs": { +# "type": "object", +# "properties": { +# "param1": {"type": "string"} +# } +# }, +# "outputs": { +# "type": "object", +# "properties": { +# "result": {"type": "string"} +# } +# }, +# "tags": [], +# "tool_provider": { +# "provider_type": "udp", +# "name": "test_udp_provider", +# "host": "localhost", +# "port": udp_provider.port +# } +# } +# ] +# } + +# mock_udp_server.set_response('{"type": "utcp"}', discovery_response) +# print(f"Mock UDP server port: {mock_udp_server.port}") +# print(f"UDP provider port: {udp_provider.port}") + +# # Register the provider +# tools = await udp_transport.register_tool_provider(udp_provider) + +# # Verify tools were returned +# assert len(tools) == 1 +# assert tools[0].name == "test_tool" +# assert tools[0].description == "Test tool" + +# # Verify logger was called +# logger.assert_called() + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_empty_response(udp_transport, udp_provider, mock_udp_server): +# """Test registering a tool provider with empty response.""" +# # Set up empty discovery response +# mock_udp_server.set_response('{"type": "utcp"}', {"tools": []}) + +# # Register the provider +# tools = await udp_transport.register_tool_provider(udp_provider) + +# # Verify no tools were returned +# assert len(tools) == 0 + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_invalid_json(udp_transport, udp_provider, mock_udp_server): +# """Test registering a tool provider with invalid JSON response.""" +# # Set up invalid JSON response +# mock_udp_server.set_response('{"type": "utcp"}', "invalid json") + +# # Register the provider +# tools = await udp_transport.register_tool_provider(udp_provider) + +# # Verify no tools were returned due to JSON error +# assert len(tools) == 0 + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_invalid_provider_type(udp_transport): +# """Test registering a non-UDP provider raises ValueError.""" +# from utcp.shared.provider import HttpProvider + +# http_provider = HttpProvider( +# name="test_http_provider", +# url="http://example.com" +# ) + +# with pytest.raises(ValueError, match="UDPTransport can only be used with UDPProvider"): +# await udp_transport.register_tool_provider(http_provider) + + +# # Test deregister_tool_provider +# @pytest.mark.asyncio +# async def test_deregister_tool_provider(udp_transport, udp_provider): +# """Test deregistering a tool provider (should be a no-op).""" +# # This should not raise any exceptions +# await udp_transport.deregister_tool_provider(udp_provider) + + +# @pytest.mark.asyncio +# async def test_deregister_tool_provider_invalid_type(udp_transport): +# """Test deregistering a non-UDP provider raises ValueError.""" +# from utcp.shared.provider import HttpProvider + +# http_provider = HttpProvider( +# name="test_http_provider", +# url="http://example.com" +# ) + +# with pytest.raises(ValueError, match="UDPTransport can only be used with UDPProvider"): +# await udp_transport.deregister_tool_provider(http_provider) + + +# # Test call_tool with JSON format +# @pytest.mark.asyncio +# async def test_call_tool_json_format(udp_transport, udp_provider, mock_udp_server): +# """Test calling a tool with JSON format.""" +# # Set up tool call response +# arguments = {"param1": "value1", "param2": 42} +# expected_message = json.dumps(arguments) +# response = {"result": "success", "data": "processed"} + +# mock_udp_server.set_response(expected_message, response) + +# # Call the tool +# result = await udp_transport.call_tool("test_tool", arguments, udp_provider) + +# # Verify response +# assert result == json.dumps(response) +# assert mock_udp_server.call_count >= 1 + + +# @pytest.mark.asyncio +# async def test_call_tool_text_template_format(udp_transport, text_template_provider, mock_udp_server): +# """Test calling a tool with text template format.""" +# # Set up tool call response +# arguments = {"action": "get", "value": "data123"} +# expected_message = "COMMAND get data123" # Template substitution +# response = "SUCCESS: data123 retrieved" + +# mock_udp_server.set_response(expected_message, response) + +# # Call the tool +# result = await udp_transport.call_tool("test_tool", arguments, text_template_provider) + +# # Verify response +# assert result == response +# assert mock_udp_server.call_count >= 1 + + +# @pytest.mark.asyncio +# async def test_call_tool_text_format_no_template(udp_transport, mock_udp_server): +# """Test calling a tool with text format but no template.""" +# provider = UDPProvider( +# name="test_provider", +# host=mock_udp_server.host, +# port=mock_udp_server.port, +# request_data_format="text", +# request_data_template=None, # No template +# response_byte_format="utf-8", +# number_of_response_datagrams=1 # Expect 1 response +# ) + +# # Set up tool call response +# arguments = {"param1": "value1", "param2": "value2"} +# expected_message = "value1 value2" # Fallback format +# response = "OK" + +# mock_udp_server.set_response(expected_message, response) + +# # Call the tool +# result = await udp_transport.call_tool("test_tool", arguments, provider) + +# # Verify response +# assert result == response + + +# @pytest.mark.asyncio +# async def test_call_tool_raw_bytes_response(udp_transport, raw_bytes_provider, mock_udp_server): +# """Test calling a tool that returns raw bytes.""" +# # Set up tool call response with raw bytes +# arguments = {"param1": "value1"} +# expected_message = json.dumps(arguments) +# raw_response = b"\x01\x02\x03\x04binary_data" + +# mock_udp_server.set_response(expected_message, raw_response) + +# # Call the tool +# result = await udp_transport.call_tool("test_tool", arguments, raw_bytes_provider) + +# # Verify response is raw bytes +# assert isinstance(result, bytes) +# assert result == raw_response + + +# @pytest.mark.asyncio +# async def test_call_tool_invalid_provider_type(udp_transport): +# """Test calling a tool with non-UDP provider raises ValueError.""" +# from utcp.shared.provider import HttpProvider + +# http_provider = HttpProvider( +# name="test_http_provider", +# url="http://example.com" +# ) + +# with pytest.raises(ValueError, match="UDPTransport can only be used with UDPProvider"): +# await udp_transport.call_tool("test_tool", {"param": "value"}, http_provider) + + +# # Test multi-datagram support +# @pytest.mark.asyncio +# async def test_call_tool_multiple_datagrams(udp_transport, multi_datagram_provider, mock_udp_server): +# """Test calling a tool that expects multiple response datagrams.""" +# # This test is complex because we need to simulate multiple UDP responses +# # For now, let's test that the transport handles the configuration correctly + +# # Mock the _send_udp_message method to simulate multiple datagram responses +# with patch.object(udp_transport, '_send_udp_message') as mock_send: +# mock_send.return_value = "part1part2part3" # Concatenated response + +# arguments = {"param1": "value1"} +# result = await udp_transport.call_tool("test_tool", arguments, multi_datagram_provider) + +# # Verify the method was called with correct parameters +# mock_send.assert_called_once_with( +# multi_datagram_provider.host, +# multi_datagram_provider.port, +# json.dumps(arguments), +# multi_datagram_provider.timeout / 1000.0, +# 3, # number_of_response_datagrams +# "utf-8" # response_byte_format +# ) + +# assert result == "part1part2part3" + + +# # Test _send_udp_message method directly +# @pytest.mark.asyncio +# async def test_send_udp_message_single_datagram(udp_transport, mock_udp_server): +# """Test sending a UDP message and receiving a single response.""" +# # Set up response +# message = "test message" +# response = "test response" +# mock_udp_server.set_response(message, response) + +# # Send message +# result = await udp_transport._send_udp_message( +# mock_udp_server.host, +# mock_udp_server.port, +# message, +# timeout=5.0, +# num_response_datagrams=1, +# response_encoding="utf-8" +# ) + +# # Verify response +# assert result == response + + +# @pytest.mark.asyncio +# async def test_send_udp_message_raw_bytes(udp_transport, mock_udp_server): +# """Test sending a UDP message and receiving raw bytes.""" +# # Set up binary response +# message = "test message" +# response = b"\x01\x02\x03binary" +# mock_udp_server.set_response(message, response) + +# # Send message with no encoding (raw bytes) +# result = await udp_transport._send_udp_message( +# mock_udp_server.host, +# mock_udp_server.port, +# message, +# timeout=5.0, +# num_response_datagrams=1, +# response_encoding=None +# ) + +# # Verify response is bytes +# assert isinstance(result, bytes) +# assert result == response + + +# @pytest.mark.asyncio +# async def test_send_udp_message_timeout(): +# """Test UDP message timeout handling.""" +# udp_transport = UDPTransport() + +# # Try to send to a non-existent server (should timeout) +# with pytest.raises(Exception): # Should raise socket timeout or connection error +# await udp_transport._send_udp_message( +# "127.0.0.1", +# 99999, # Non-existent port +# "test message", +# timeout=0.1, # Very short timeout +# num_response_datagrams=1, +# response_encoding="utf-8" +# ) + + +# # Test _format_tool_call_message method +# def test_format_tool_call_message_json(udp_transport): +# """Test formatting tool call message with JSON format.""" +# provider = UDPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="json" +# ) + +# arguments = {"param1": "value1", "param2": 42} +# result = udp_transport._format_tool_call_message(arguments, provider) + +# # Should return JSON string +# assert result == json.dumps(arguments) + +# # Verify it's valid JSON +# parsed = json.loads(result) +# assert parsed == arguments + + +# def test_format_tool_call_message_text_with_template(udp_transport): +# """Test formatting tool call message with text template.""" +# provider = UDPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="text", +# request_data_template="ACTION UTCP_ARG_cmd_UTCP_ARG PARAM UTCP_ARG_value_UTCP_ARG" +# ) + +# arguments = {"cmd": "get", "value": "data123"} +# result = udp_transport._format_tool_call_message(arguments, provider) + +# # Should substitute placeholders +# assert result == "ACTION get PARAM data123" + + +# def test_format_tool_call_message_text_with_complex_values(udp_transport): +# """Test formatting tool call message with complex values in template.""" +# provider = UDPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="text", +# request_data_template="DATA UTCP_ARG_obj_UTCP_ARG" +# ) + +# arguments = {"obj": {"nested": "value", "number": 123}} +# result = udp_transport._format_tool_call_message(arguments, provider) + +# # Should JSON-serialize complex values +# assert result == 'DATA {"nested": "value", "number": 123}' + + +# def test_format_tool_call_message_text_no_template(udp_transport): +# """Test formatting tool call message with text format but no template.""" +# provider = UDPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="text", +# request_data_template=None +# ) + +# arguments = {"param1": "value1", "param2": "value2"} +# result = udp_transport._format_tool_call_message(arguments, provider) + +# # Should use fallback format (space-separated values) +# assert result == "value1 value2" + + +# def test_format_tool_call_message_default_to_json(udp_transport): +# """Test formatting tool call message defaults to JSON for unknown format.""" +# # Create a provider with valid format first +# provider = UDPProvider( +# name="test", +# host="localhost", +# port=1234, +# request_data_format="json" +# ) + +# # Manually set an invalid format to test the fallback behavior +# provider.request_data_format = "unknown" # Invalid format + +# arguments = {"param1": "value1"} +# result = udp_transport._format_tool_call_message(arguments, provider) + +# # Should default to JSON +# assert result == json.dumps(arguments) + + +# # Test error handling and edge cases +# @pytest.mark.asyncio +# async def test_call_tool_server_error(udp_transport, udp_provider, mock_udp_server): +# """Test handling server errors during tool calls.""" +# # Don't set any response, so the server will return an error +# arguments = {"param1": "value1"} + +# # Call the tool - should get the default error response +# result = await udp_transport.call_tool("test_tool", arguments, udp_provider) + +# # Should receive the default error message +# assert '{"error": "unknown_message"}' in result + + +# @pytest.mark.asyncio +# async def test_register_tool_provider_malformed_tool(udp_transport, udp_provider, mock_udp_server): +# """Test registering provider with malformed tool definition.""" +# # Set up discovery response with invalid tool +# discovery_response = { +# "tools": [ +# { +# "name": "test_tool", +# # Missing required fields like inputs, outputs, tool_provider +# } +# ] +# } + +# mock_udp_server.set_response('{"type": "utcp"}', discovery_response) + +# # Register the provider - should handle invalid tool gracefully +# tools = await udp_transport.register_tool_provider(udp_provider) + +# # Should return empty list due to invalid tool definition +# assert len(tools) == 0 + + +# # Test logging functionality +# @pytest.mark.asyncio +# async def test_logging_calls(udp_transport, udp_provider, mock_udp_server, logger): +# """Test that logging functions are called appropriately.""" +# # Set up discovery response +# discovery_response = {"tools": []} +# mock_udp_server.set_response('{"type": "utcp"}', discovery_response) + +# # Register provider +# await udp_transport.register_tool_provider(udp_provider) + +# # Verify logger was called +# logger.assert_called() + +# # Call tool +# mock_udp_server.set_response('{}', {"result": "test"}) +# await udp_transport.call_tool("test_tool", {}, udp_provider) + +# # Logger should have been called multiple times +# assert logger.call_count > 1 diff --git a/plugins/communication_protocols/socket/pyproject.toml b/plugins/communication_protocols/socket/pyproject.toml new file mode 100644 index 0000000..fde3bd6 --- /dev/null +++ b/plugins/communication_protocols/socket/pyproject.toml @@ -0,0 +1,39 @@ +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" + +[project] +name = "utcp-socket" +version = "1.0.0" +authors = [ + { name = "UTCP Contributors" }, +] +description = "Universal Tool Calling Protocol (UTCP) client library for Python" +readme = "README.md" +requires-python = ">=3.10" +dependencies = [ + "pydantic>=2.0", + "utcp>=1.0" +] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "Programming Language :: Python :: 3", + "Operating System :: OS Independent", +] +license = "MPL-2.0" + +[project.optional-dependencies] +dev = [ + "build", + "pytest", + "pytest-asyncio", + "pytest-cov", + "coverage", + "twine", +] + +[project.urls] +Homepage = "https://utcp.io" +Source = "https://github.com/universal-tool-calling-protocol/python-utcp" +Issues = "https://github.com/universal-tool-calling-protocol/python-utcp/issues" \ No newline at end of file diff --git a/plugins/communication_protocols/socket/src/utcp_socket/__init__.py b/plugins/communication_protocols/socket/src/utcp_socket/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/plugins/communication_protocols/socket/src/utcp_socket/tcp_call_template.py b/plugins/communication_protocols/socket/src/utcp_socket/tcp_call_template.py new file mode 100644 index 0000000..157e43c --- /dev/null +++ b/plugins/communication_protocols/socket/src/utcp_socket/tcp_call_template.py @@ -0,0 +1,79 @@ +from utcp.data.call_template import CallTemplate +from typing import Optional, Literal +from pydantic import Field + +class TCPProvider(CallTemplate): + """Provider configuration for raw TCP socket tools. + + Enables direct communication with TCP servers using custom protocols. + Supports flexible request formatting, response decoding, and multiple + framing strategies for message boundaries. + + Request Data Handling: + - 'json' format: Arguments formatted as JSON object + - 'text' format: Template-based with UTCP_ARG_argname_UTCP_ARG placeholders + + Response Data Handling: + - If response_byte_format is None: Returns raw bytes + - If response_byte_format is encoding string: Decodes bytes to text + + TCP Stream Framing Options: + 1. Length-prefix: Set framing_strategy='length_prefix' + length_prefix_bytes + 2. Delimiter-based: Set framing_strategy='delimiter' + message_delimiter + 3. Fixed-length: Set framing_strategy='fixed_length' + fixed_message_length + 4. Stream-based: Set framing_strategy='stream' (reads until connection closes) + + Attributes: + call_template_type: Always "tcp" for TCP providers. + host: The hostname or IP address of the TCP server. + port: The port number of the TCP server. + request_data_format: Format for request data ('json' or 'text'). + request_data_template: Template string for 'text' format with placeholders. + response_byte_format: Encoding for response decoding (None for raw bytes). + framing_strategy: Method for detecting message boundaries. + length_prefix_bytes: Number of bytes for length prefix (1, 2, 4, or 8). + length_prefix_endian: Byte order for length prefix ('big' or 'little'). + message_delimiter: Delimiter string for message boundaries. + fixed_message_length: Fixed length in bytes for each message. + max_response_size: Maximum bytes to read for stream-based framing. + timeout: Connection timeout in milliseconds. + auth: Always None - TCP providers don't support authentication. + """ + + call_template_type: Literal["tcp"] = "tcp" + host: str + port: int + request_data_format: Literal["json", "text"] = "json" + request_data_template: Optional[str] = None + response_byte_format: Optional[str] = Field(default="utf-8", description="Encoding to decode response bytes. If None, returns raw bytes.") + # TCP Framing Strategy + framing_strategy: Literal["length_prefix", "delimiter", "fixed_length", "stream"] = Field( + default="stream", + description="Strategy for framing TCP messages" + ) + # Length-prefix framing options + length_prefix_bytes: Literal[1, 2, 4, 8] = Field( + default=4, + description="Number of bytes for length prefix (1, 2, 4, or 8). Used with 'length_prefix' framing." + ) + length_prefix_endian: Literal["big", "little"] = Field( + default="big", + description="Byte order for length prefix. Used with 'length_prefix' framing." + ) + # Delimiter-based framing options + message_delimiter: str = Field( + default='\x00', + description="Delimiter to detect end of TCP response (e.g., '\\n', '\\r\\n', '\\x00'). Used with 'delimiter' framing." + ) + # Fixed-length framing options + fixed_message_length: Optional[int] = Field( + default=None, + description="Fixed length of each message in bytes. Used with 'fixed_length' framing." + ) + # Stream-based options + max_response_size: int = Field( + default=65536, + description="Maximum bytes to read from TCP stream. Used with 'stream' framing." + ) + timeout: int = 30000 + auth: None = None diff --git a/src/utcp/client/transport_interfaces/tcp_transport.py b/plugins/communication_protocols/socket/src/utcp_socket/tcp_communication_protocol.py similarity index 96% rename from src/utcp/client/transport_interfaces/tcp_transport.py rename to plugins/communication_protocols/socket/src/utcp_socket/tcp_communication_protocol.py index 216c3f4..95d0e5f 100644 --- a/src/utcp/client/transport_interfaces/tcp_transport.py +++ b/plugins/communication_protocols/socket/src/utcp_socket/tcp_communication_protocol.py @@ -5,7 +5,6 @@ """ import asyncio import json -import logging import socket import struct from typing import Dict, Any, List, Optional, Callable, Union @@ -13,7 +12,9 @@ from utcp.client.client_transport_interface import ClientTransportInterface from utcp.shared.provider import Provider, TCPProvider from utcp.shared.tool import Tool +import logging +logger = logging.getLogger(__name__) class TCPTransport(ClientTransportInterface): """Transport implementation for TCP-based tool providers. @@ -42,30 +43,30 @@ def _log_info(self, message: str): def _log_error(self, message: str): """Log error messages.""" - logging.error(f"[TCPTransport Error] {message}") + logger.error(f"[TCPTransport Error] {message}") def _format_tool_call_message( self, - arguments: Dict[str, Any], + tool_args: Dict[str, Any], provider: TCPProvider ) -> str: """Format a tool call message based on provider configuration. Args: - arguments: Arguments for the tool call + tool_args: Arguments for the tool call provider: The TCPProvider with formatting configuration Returns: Formatted message string """ if provider.request_data_format == "json": - return json.dumps(arguments) + return json.dumps(tool_args) elif provider.request_data_format == "text": # Use template-based formatting if provider.request_data_template is not None and provider.request_data_template != "": message = provider.request_data_template # Replace placeholders with argument values - for arg_name, arg_value in arguments.items(): + for arg_name, arg_value in tool_args.items(): placeholder = f"UTCP_ARG_{arg_name}_UTCP_ARG" if isinstance(arg_value, str): message = message.replace(placeholder, arg_value) @@ -74,10 +75,10 @@ def _format_tool_call_message( return message else: # Fallback to simple key=value format - return " ".join([str(v) for k, v in arguments.items()]) + return " ".join([str(v) for k, v in tool_args.items()]) else: # Default to JSON format - return json.dumps(arguments) + return json.dumps(tool_args) def _encode_message_with_framing(self, message: str, provider: TCPProvider) -> bytes: """Encode message with appropriate TCP framing. @@ -367,14 +368,14 @@ async def deregister_tool_provider(self, manual_provider: Provider) -> None: self._log_info(f"Deregistering TCP provider '{manual_provider.name}' (no-op)") - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> Any: + async def call_tool(self, tool_name: str, tool_args: Dict[str, Any], tool_provider: Provider) -> Any: """Call a TCP tool. Sends a tool call message to the TCP provider and returns the response. Args: tool_name: Name of the tool to call - arguments: Arguments for the tool call + tool_args: Arguments for the tool call tool_provider: The TCPProvider containing the tool Returns: @@ -389,7 +390,7 @@ async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provid self._log_info(f"Calling TCP tool '{tool_name}' on provider '{tool_provider.name}'") try: - tool_call_message = self._format_tool_call_message(arguments, tool_provider) + tool_call_message = self._format_tool_call_message(tool_args, tool_provider) response = await self._send_tcp_message( tool_provider.host, diff --git a/plugins/communication_protocols/socket/src/utcp_socket/udp_call_template.py b/plugins/communication_protocols/socket/src/utcp_socket/udp_call_template.py new file mode 100644 index 0000000..4c704da --- /dev/null +++ b/plugins/communication_protocols/socket/src/utcp_socket/udp_call_template.py @@ -0,0 +1,40 @@ +from utcp.data.call_template import CallTemplate +from typing import Optional, Literal +from pydantic import Field + +class UDPProvider(CallTemplate): + """Provider configuration for UDP (User Datagram Protocol) socket tools. + + Enables communication with UDP servers using the connectionless UDP protocol. + Supports flexible request formatting, response decoding, and multi-datagram + response handling. + + Request Data Handling: + - 'json' format: Arguments formatted as JSON object + - 'text' format: Template-based with UTCP_ARG_argname_UTCP_ARG placeholders + + Response Data Handling: + - If response_byte_format is None: Returns raw bytes + - If response_byte_format is encoding string: Decodes bytes to text + + Attributes: + call_template_type: Always "udp" for UDP providers. + host: The hostname or IP address of the UDP server. + port: The port number of the UDP server. + number_of_response_datagrams: Expected number of response datagrams (0 for no response). + request_data_format: Format for request data ('json' or 'text'). + request_data_template: Template string for 'text' format with placeholders. + response_byte_format: Encoding for response decoding (None for raw bytes). + timeout: Request timeout in milliseconds. + auth: Always None - UDP providers don't support authentication. + """ + + call_template_type: Literal["udp"] = "udp" + host: str + port: int + number_of_response_datagrams: int = 1 + request_data_format: Literal["json", "text"] = "json" + request_data_template: Optional[str] = None + response_byte_format: Optional[str] = Field(default="utf-8", description="Encoding to decode response bytes. If None, returns raw bytes.") + timeout: int = 30000 + auth: None = None diff --git a/src/utcp/client/transport_interfaces/udp_transport.py b/plugins/communication_protocols/socket/src/utcp_socket/udp_communication_protocol.py similarity index 92% rename from src/utcp/client/transport_interfaces/udp_transport.py rename to plugins/communication_protocols/socket/src/utcp_socket/udp_communication_protocol.py index 16228e3..8d4d404 100644 --- a/src/utcp/client/transport_interfaces/udp_transport.py +++ b/plugins/communication_protocols/socket/src/utcp_socket/udp_communication_protocol.py @@ -5,14 +5,16 @@ """ import asyncio import json -import logging import socket +import traceback from typing import Dict, Any, List, Optional, Callable, Union from utcp.client.client_transport_interface import ClientTransportInterface from utcp.shared.provider import Provider, UDPProvider from utcp.shared.tool import Tool +import logging +logger = logging.getLogger(__name__) class UDPTransport(ClientTransportInterface): """Transport implementation for UDP-based tool providers. @@ -42,30 +44,30 @@ def _log_info(self, message: str): def _log_error(self, message: str): """Log error messages.""" - logging.error(f"[UDPTransport Error] {message}") + logger.error(f"[UDPTransport Error] {message}") def _format_tool_call_message( self, - arguments: Dict[str, Any], + tool_args: Dict[str, Any], provider: UDPProvider ) -> str: """Format a tool call message based on provider configuration. Args: - arguments: Arguments for the tool call + tool_args: Arguments for the tool call provider: The UDPProvider with formatting configuration Returns: Formatted message string """ if provider.request_data_format == "json": - return json.dumps(arguments) + return json.dumps(tool_args) elif provider.request_data_format == "text": # Use template-based formatting if provider.request_data_template is not None and provider.request_data_template != "": message = provider.request_data_template # Replace placeholders with argument values - for arg_name, arg_value in arguments.items(): + for arg_name, arg_value in tool_args.items(): placeholder = f"UTCP_ARG_{arg_name}_UTCP_ARG" if isinstance(arg_value, str): message = message.replace(placeholder, arg_value) @@ -74,10 +76,10 @@ def _format_tool_call_message( return message else: # Fallback to simple key=value format - return " ".join([str(v) for k, v in arguments.items()]) + return " ".join([str(v) for k, v in tool_args.items()]) else: # Default to JSON format - return json.dumps(arguments) + return json.dumps(tool_args) async def _send_udp_message( self, @@ -176,10 +178,10 @@ def _send_and_receive(): return combined_bytes except TimeoutError as e: - self._log_error(str(e)) - raise asyncio.TimeoutError(str(e)) + self._log_error(traceback.format_exc()) + raise asyncio.TimeoutError(traceback.format_exc()) except Exception as e: - self._log_error(f"Error sending UDP message: {e}") + self._log_error(f"Error sending UDP message: {traceback.format_exc()}") raise async def _send_udp_no_response(self, host: str, port: int, message: str) -> None: @@ -197,7 +199,7 @@ def _send_only(): loop = asyncio.get_event_loop() await loop.run_in_executor(None, _send_only) except Exception as e: - self._log_error(f"Error sending UDP message (no response): {e}") + self._log_error(f"Error sending UDP message (no response): {traceback.format_exc()}") raise async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: @@ -255,7 +257,7 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: tool = Tool(**tool_data) tools.append(tool) except Exception as e: - self._log_error(f"Invalid tool definition in UDP provider '{manual_provider.name}': {e}") + self._log_error(f"Invalid tool definition in UDP provider '{manual_provider.name}': {traceback.format_exc()}") continue self._log_info(f"Discovered {len(tools)} tools from UDP provider '{manual_provider.name}'") @@ -265,11 +267,11 @@ async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: return [] except json.JSONDecodeError as e: - self._log_error(f"Invalid JSON response from UDP provider '{manual_provider.name}': {e}") + self._log_error(f"Invalid JSON response from UDP provider '{manual_provider.name}': {traceback.format_exc()}") return [] except Exception as e: - self._log_error(f"Error registering UDP provider '{manual_provider.name}': {e}") + self._log_error(f"Error registering UDP provider '{manual_provider.name}': {traceback.format_exc()}") return [] async def deregister_tool_provider(self, manual_provider: Provider) -> None: @@ -285,7 +287,7 @@ async def deregister_tool_provider(self, manual_provider: Provider) -> None: self._log_info(f"Deregistering UDP provider '{manual_provider.name}' (no-op)") - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> Any: + async def call_tool(self, tool_name: str, tool_args: Dict[str, Any], tool_provider: Provider) -> Any: """Call a UDP tool. Sends a tool call message to the UDP provider and returns the response. @@ -307,7 +309,7 @@ async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provid self._log_info(f"Calling UDP tool '{tool_name}' on provider '{tool_provider.name}'") try: - tool_call_message = self._format_tool_call_message(arguments, tool_provider) + tool_call_message = self._format_tool_call_message(tool_args, tool_provider) response = await self._send_udp_message( tool_provider.host, @@ -320,5 +322,5 @@ async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provid return response except Exception as e: - self._log_error(f"Error calling UDP tool '{tool_name}': {e}") + self._log_error(f"Error calling UDP tool '{tool_name}': {traceback.format_exc()}") raise diff --git a/plugins/communication_protocols/text/pyproject.toml b/plugins/communication_protocols/text/pyproject.toml new file mode 100644 index 0000000..d5b3993 --- /dev/null +++ b/plugins/communication_protocols/text/pyproject.toml @@ -0,0 +1,44 @@ +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" + +[project] +name = "utcp-text" +version = "1.0.0" +authors = [ + { name = "UTCP Contributors" }, +] +description = "Universal Tool Calling Protocol (UTCP) client library for Python" +readme = "README.md" +requires-python = ">=3.10" +dependencies = [ + "pydantic>=2.0", + "pyyaml>=6.0", + "utcp>=1.0", + "utcp-http>=1.0" +] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "Programming Language :: Python :: 3", + "Operating System :: OS Independent", +] +license = "MPL-2.0" + +[project.optional-dependencies] +dev = [ + "build", + "pytest", + "pytest-asyncio", + "pytest-cov", + "coverage", + "twine", +] + +[project.urls] +Homepage = "https://utcp.io" +Source = "https://github.com/universal-tool-calling-protocol/python-utcp" +Issues = "https://github.com/universal-tool-calling-protocol/python-utcp/issues" + +[project.entry-points."utcp.plugins"] +text = "utcp_text:register" \ No newline at end of file diff --git a/plugins/communication_protocols/text/src/utcp_text/__init__.py b/plugins/communication_protocols/text/src/utcp_text/__init__.py new file mode 100644 index 0000000..56fcf59 --- /dev/null +++ b/plugins/communication_protocols/text/src/utcp_text/__init__.py @@ -0,0 +1,15 @@ +"""Text Communication Protocol plugin for UTCP.""" + +from utcp.plugins.discovery import register_communication_protocol, register_call_template +from utcp_text.text_communication_protocol import TextCommunicationProtocol +from utcp_text.text_call_template import TextCallTemplate, TextCallTemplateSerializer + +def register(): + register_communication_protocol("text", TextCommunicationProtocol()) + register_call_template("text", TextCallTemplateSerializer()) + +__all__ = [ + "TextCommunicationProtocol", + "TextCallTemplate", + "TextCallTemplateSerializer", +] diff --git a/plugins/communication_protocols/text/src/utcp_text/text_call_template.py b/plugins/communication_protocols/text/src/utcp_text/text_call_template.py new file mode 100644 index 0000000..ee9af09 --- /dev/null +++ b/plugins/communication_protocols/text/src/utcp_text/text_call_template.py @@ -0,0 +1,36 @@ +from typing import Literal +from pydantic import Field + +from utcp.data.call_template import CallTemplate +from utcp.interfaces.serializer import Serializer +from utcp.exceptions import UtcpSerializerValidationError +import traceback + +class TextCallTemplate(CallTemplate): + """Call template for text file-based manuals and tools. + + Reads UTCP manuals or tool definitions from local JSON/YAML files. Useful for + static tool configurations or environments where manuals are distributed as files. + + Attributes: + call_template_type: Always "text" for text file call templates. + file_path: Path to the file containing the UTCP manual or tool definitions. + auth: Always None - text call templates don't support authentication. + """ + + call_template_type: Literal["text"] = "text" + file_path: str = Field(..., description="The path to the file containing the UTCP manual or tool definitions.") + auth: None = None + + +class TextCallTemplateSerializer(Serializer[TextCallTemplate]): + """Serializer for TextCallTemplate.""" + + def to_dict(self, obj: TextCallTemplate) -> dict: + return obj.model_dump() + + def validate_dict(self, obj: dict) -> TextCallTemplate: + try: + return TextCallTemplate.model_validate(obj) + except Exception as e: + raise UtcpSerializerValidationError("Invalid TextCallTemplate: " + traceback.format_exc()) diff --git a/plugins/communication_protocols/text/src/utcp_text/text_communication_protocol.py b/plugins/communication_protocols/text/src/utcp_text/text_communication_protocol.py new file mode 100644 index 0000000..2db1b49 --- /dev/null +++ b/plugins/communication_protocols/text/src/utcp_text/text_communication_protocol.py @@ -0,0 +1,121 @@ +""" +Text communication protocol for UTCP client. + +This protocol reads UTCP manuals (or OpenAPI specs) from local files to register +tools. It does not maintain any persistent connections. +""" +import json +import yaml +from pathlib import Path +from typing import Dict, Any, Optional, AsyncGenerator, TYPE_CHECKING + +from utcp.interfaces.communication_protocol import CommunicationProtocol +from utcp.data.call_template import CallTemplate +from utcp.data.utcp_manual import UtcpManual, UtcpManualSerializer +from utcp.data.register_manual_response import RegisterManualResult +from utcp_http.openapi_converter import OpenApiConverter +from utcp_text.text_call_template import TextCallTemplate +import traceback + +if TYPE_CHECKING: + from utcp.utcp_client import UtcpClient + +import logging + +logger = logging.getLogger(__name__) + +class TextCommunicationProtocol(CommunicationProtocol): + """Communication protocol for file-based UTCP manuals and tools.""" + + def _log_info(self, message: str) -> None: + logger.info(f"[TextCommunicationProtocol] {message}") + + def _log_error(self, message: str) -> None: + logger.error(f"[TextCommunicationProtocol Error] {message}") + + async def register_manual(self, caller: 'UtcpClient', manual_call_template: CallTemplate) -> RegisterManualResult: + """Register a text manual and return its tools as a UtcpManual.""" + if not isinstance(manual_call_template, TextCallTemplate): + raise ValueError("TextCommunicationProtocol requires a TextCallTemplate") + + file_path = Path(manual_call_template.file_path) + if not file_path.is_absolute() and caller.root_dir: + file_path = Path(caller.root_dir) / file_path + + self._log_info(f"Reading manual from '{file_path}'") + + try: + if not file_path.exists(): + raise FileNotFoundError(f"Manual file not found: {file_path}") + + with open(file_path, "r", encoding="utf-8") as f: + file_content = f.read() + + # Parse based on extension + data: Any + if file_path.suffix.lower() in [".yaml", ".yml"]: + data = yaml.safe_load(file_content) + else: + data = json.loads(file_content) + + utcp_manual: UtcpManual + if isinstance(data, dict) and ("openapi" in data or "swagger" in data or "paths" in data): + self._log_info("Detected OpenAPI specification. Converting to UTCP manual.") + converter = OpenApiConverter(data, spec_url=file_path.as_uri(), call_template_name=manual_call_template.name) + utcp_manual = converter.convert() + else: + # Try to validate as UTCP manual directly + utcp_manual = UtcpManualSerializer().validate_dict(data) + + self._log_info(f"Loaded {len(utcp_manual.tools)} tools from '{file_path}'") + return RegisterManualResult( + manual_call_template=manual_call_template, + manual=utcp_manual, + success=True, + errors=[], + ) + + except (json.JSONDecodeError, yaml.YAMLError) as e: + self._log_error(f"Failed to parse manual '{file_path}': {traceback.format_exc()}") + return RegisterManualResult( + manual_call_template=manual_call_template, + manual=UtcpManual(tools=[]), + success=False, + errors=[traceback.format_exc()], + ) + except Exception as e: + self._log_error(f"Unexpected error reading manual '{file_path}': {traceback.format_exc()}") + return RegisterManualResult( + manual_call_template=manual_call_template, + manual=UtcpManual(tools=[]), + success=False, + errors=[traceback.format_exc()], + ) + + async def deregister_manual(self, caller: 'UtcpClient', manual_call_template: CallTemplate) -> None: + """Deregister a text manual (no-op).""" + if isinstance(manual_call_template, TextCallTemplate): + self._log_info(f"Deregistering text manual '{manual_call_template.name}' (no-op)") + + async def call_tool(self, caller: 'UtcpClient', tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: + """Call a tool: for text templates, return file content from the configured path.""" + if not isinstance(tool_call_template, TextCallTemplate): + raise ValueError("TextCommunicationProtocol requires a TextCallTemplate for tool calls") + + file_path = Path(tool_call_template.file_path) + if not file_path.is_absolute() and caller.root_dir: + file_path = Path(caller.root_dir) / file_path + + self._log_info(f"Reading content from '{file_path}' for tool '{tool_name}'") + + if not file_path.exists(): + raise FileNotFoundError(f"File not found: {file_path}") + + with open(file_path, "r", encoding="utf-8") as f: + content = f.read() + return content + + async def call_tool_streaming(self, caller: 'UtcpClient', tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> AsyncGenerator[Any, None]: + """Streaming variant: yields the full content as a single chunk.""" + result = await self.call_tool(caller, tool_name, tool_args, tool_call_template) + yield result diff --git a/plugins/communication_protocols/text/tests/test_text_communication_protocol.py b/plugins/communication_protocols/text/tests/test_text_communication_protocol.py new file mode 100644 index 0000000..0b7dffb --- /dev/null +++ b/plugins/communication_protocols/text/tests/test_text_communication_protocol.py @@ -0,0 +1,358 @@ +""" +Tests for the Text communication protocol (file-based) implementation. +""" +import json +import tempfile +from pathlib import Path +import pytest +import pytest_asyncio +from unittest.mock import Mock + +from utcp_text.text_communication_protocol import TextCommunicationProtocol +from utcp_text.text_call_template import TextCallTemplate +from utcp.data.call_template import CallTemplate +from utcp.data.register_manual_response import RegisterManualResult +from utcp.utcp_client import UtcpClient + +@pytest_asyncio.fixture +async def text_protocol() -> TextCommunicationProtocol: + """Provides a TextCommunicationProtocol instance.""" + yield TextCommunicationProtocol() + + +@pytest_asyncio.fixture +def mock_utcp_client(tmp_path: Path) -> Mock: + """Provides a mock UtcpClient with a root_dir.""" + client = Mock(spec=UtcpClient) + client.root_dir = tmp_path + return client + + +@pytest_asyncio.fixture +def sample_utcp_manual(): + """Sample UTCP manual with multiple tools (new UTCP format).""" + return { + "utcp_version": "1.0.0", + "manual_version": "1.0.0", + "tools": [ + { + "name": "calculator", + "description": "Performs basic arithmetic operations", + "inputs": { + "type": "object", + "properties": { + "operation": { + "type": "string", + "enum": ["add", "subtract", "multiply", "divide"] + }, + "a": {"type": "number"}, + "b": {"type": "number"} + }, + "required": ["operation", "a", "b"] + }, + "outputs": { + "type": "object", + "properties": { + "result": {"type": "number"} + } + }, + "tags": ["math", "arithmetic"], + "tool_call_template": { + "call_template_type": "text", + "name": "test-text-call-template", + "file_path": "dummy.json" + } + }, + { + "name": "string_utils", + "description": "String manipulation utilities", + "inputs": { + "type": "object", + "properties": { + "text": {"type": "string"}, + "operation": { + "type": "string", + "enum": ["uppercase", "lowercase", "reverse"] + } + }, + "required": ["text", "operation"] + }, + "outputs": { + "type": "object", + "properties": { + "result": {"type": "string"} + } + }, + "tags": ["text", "utilities"], + "tool_call_template": { + "call_template_type": "text", + "name": "test-text-call-template", + "file_path": "dummy.json" + } + } + ] + } + + +@pytest_asyncio.fixture +def single_tool_definition(): + """Sample single tool definition (new UTCP format).""" + return { + "name": "echo", + "description": "Echoes back the input text", + "inputs": { + "type": "object", + "properties": { + "message": {"type": "string"} + }, + "required": ["message"] + }, + "outputs": { + "type": "object", + "properties": { + "echo": {"type": "string"} + } + }, + "tags": ["utility"], + "tool_call_template": { + "call_template_type": "text", + "name": "test-text-call-template", + "file_path": "dummy.json" + } + } + + +@pytest_asyncio.fixture +def tool_array(): + """Sample array of tool definitions (new UTCP format).""" + return [ + { + "name": "tool1", + "description": "First tool", + "inputs": {"type": "object", "properties": {}, "required": []}, + "outputs": {"type": "object", "properties": {}, "required": []}, + "tags": [], + "tool_call_template": { + "call_template_type": "text", + "name": "test-text-call-template", + "file_path": "dummy.json" + } + }, + { + "name": "tool2", + "description": "Second tool", + "inputs": {"type": "object", "properties": {}, "required": []}, + "outputs": {"type": "object", "properties": {}, "required": []}, + "tags": [], + "tool_call_template": { + "call_template_type": "text", + "name": "test-text-call-template", + "file_path": "dummy.json" + } + } + ] + + +@pytest.mark.asyncio +async def test_register_manual_with_utcp_manual( + text_protocol: TextCommunicationProtocol, sample_utcp_manual, mock_utcp_client: Mock +): + """Register a manual from a local file and validate returned tools.""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + json.dump(sample_utcp_manual, f) + temp_file = f.name + + try: + manual_template = TextCallTemplate(name="test_manual", file_path=temp_file) + result = await text_protocol.register_manual(mock_utcp_client, manual_template) + + assert isinstance(result, RegisterManualResult) + assert result.success is True + assert result.errors == [] + assert result.manual is not None + assert len(result.manual.tools) == 2 + + tool0 = result.manual.tools[0] + assert tool0.name == "calculator" + assert tool0.description == "Performs basic arithmetic operations" + assert tool0.tags == ["math", "arithmetic"] + assert tool0.tool_call_template.call_template_type == "text" + + tool1 = result.manual.tools[1] + assert tool1.name == "string_utils" + assert tool1.description == "String manipulation utilities" + assert tool1.tags == ["text", "utilities"] + assert tool1.tool_call_template.call_template_type == "text" + finally: + Path(temp_file).unlink() + + +@pytest.mark.asyncio +async def test_register_manual_with_single_tool( + text_protocol: TextCommunicationProtocol, single_tool_definition, mock_utcp_client: Mock +): + """Register a manual with a single tool definition.""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + manual = { + "utcp_version": "1.0.0", + "manual_version": "1.0.0", + "tools": [single_tool_definition], + } + json.dump(manual, f) + temp_file = f.name + + try: + manual_template = TextCallTemplate(name="single_tool_manual", file_path=temp_file) + result = await text_protocol.register_manual(mock_utcp_client, manual_template) + + assert result.success is True + assert len(result.manual.tools) == 1 + tool = result.manual.tools[0] + assert tool.name == "echo" + assert tool.description == "Echoes back the input text" + assert tool.tags == ["utility"] + assert tool.tool_call_template.call_template_type == "text" + finally: + Path(temp_file).unlink() + + +@pytest.mark.asyncio +async def test_register_manual_with_tool_array( + text_protocol: TextCommunicationProtocol, tool_array, mock_utcp_client: Mock +): + """Register a manual with an array of tool definitions.""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + manual = { + "utcp_version": "1.0.0", + "manual_version": "1.0.0", + "tools": tool_array, + } + json.dump(manual, f) + temp_file = f.name + + try: + manual_template = TextCallTemplate(name="tool_array_manual", file_path=temp_file) + result = await text_protocol.register_manual(mock_utcp_client, manual_template) + + assert result.success is True + assert len(result.manual.tools) == 2 + assert result.manual.tools[0].name == "tool1" + assert result.manual.tools[1].name == "tool2" + assert result.manual.tools[0].tool_call_template.call_template_type == "text" + assert result.manual.tools[1].tool_call_template.call_template_type == "text" + finally: + Path(temp_file).unlink() + + +@pytest.mark.asyncio +async def test_register_manual_file_not_found( + text_protocol: TextCommunicationProtocol, mock_utcp_client: Mock +): + """Registering a manual with a non-existent file should return errors.""" + manual_template = TextCallTemplate(name="missing", file_path="/path/that/does/not/exist.json") + result = await text_protocol.register_manual(mock_utcp_client, manual_template) + assert isinstance(result, RegisterManualResult) + assert result.success is False + assert result.errors + + +@pytest.mark.asyncio +async def test_register_manual_invalid_json( + text_protocol: TextCommunicationProtocol, mock_utcp_client: Mock +): + """Registering a manual with invalid JSON should return errors (no exception).""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + f.write("{ invalid json content }") + temp_file = f.name + + try: + manual_template = TextCallTemplate(name="invalid_json", file_path=temp_file) + result = await text_protocol.register_manual(mock_utcp_client, manual_template) + assert isinstance(result, RegisterManualResult) + assert result.success is False + assert result.errors + finally: + Path(temp_file).unlink() + + +@pytest.mark.asyncio +async def test_register_manual_wrong_call_template_type(text_protocol: TextCommunicationProtocol, mock_utcp_client: Mock): + """Registering with a non-Text call template should raise ValueError.""" + wrong_template = CallTemplate(call_template_type="invalid", name="wrong") + with pytest.raises(ValueError, match="requires a TextCallTemplate"): + await text_protocol.register_manual(mock_utcp_client, wrong_template) # type: ignore[arg-type] + + +@pytest.mark.asyncio +async def test_call_tool_returns_file_content( + text_protocol: TextCommunicationProtocol, sample_utcp_manual, mock_utcp_client: Mock +): + """Calling a tool returns the file content from the call template path.""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + json.dump(sample_utcp_manual, f) + temp_file = f.name + + try: + tool_template = TextCallTemplate(name="tool_call", file_path=temp_file) + + # Call a tool should return the file content + content = await text_protocol.call_tool( + mock_utcp_client, "calculator", {"operation": "add", "a": 1, "b": 2}, tool_template + ) + + # Verify we get the JSON content back as a string + assert isinstance(content, str) + # Parse it back to verify it's the same content + parsed_content = json.loads(content) + assert parsed_content == sample_utcp_manual + finally: + Path(temp_file).unlink() + + +@pytest.mark.asyncio +async def test_call_tool_wrong_call_template_type(text_protocol: TextCommunicationProtocol, mock_utcp_client: Mock): + """Calling a tool with wrong call template type should raise ValueError.""" + wrong_template = CallTemplate(call_template_type="invalid", name="wrong") + with pytest.raises(ValueError, match="requires a TextCallTemplate"): + await text_protocol.call_tool(mock_utcp_client, "some_tool", {}, wrong_template) # type: ignore[arg-type] + + +@pytest.mark.asyncio +async def test_call_tool_file_not_found(text_protocol: TextCommunicationProtocol, mock_utcp_client: Mock): + """Calling a tool when the file doesn't exist should raise FileNotFoundError.""" + tool_template = TextCallTemplate(name="missing", file_path="/path/that/does/not/exist.json") + with pytest.raises(FileNotFoundError): + await text_protocol.call_tool(mock_utcp_client, "some_tool", {}, tool_template) + + +@pytest.mark.asyncio +async def test_deregister_manual(text_protocol: TextCommunicationProtocol, sample_utcp_manual, mock_utcp_client: Mock): + """Deregistering a manual should be a no-op (no errors).""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + json.dump(sample_utcp_manual, f) + temp_file = f.name + + try: + manual_template = TextCallTemplate(name="test_manual", file_path=temp_file) + await text_protocol.deregister_manual(mock_utcp_client, manual_template) + finally: + Path(temp_file).unlink() + + +@pytest.mark.asyncio +async def test_call_tool_streaming(text_protocol: TextCommunicationProtocol, sample_utcp_manual, mock_utcp_client: Mock): + """Streaming call should yield a single chunk equal to non-streaming content.""" + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + json.dump(sample_utcp_manual, f) + temp_file = f.name + + try: + tool_template = TextCallTemplate(name="tool_call", file_path=temp_file) + # Non-streaming + content = await text_protocol.call_tool(mock_utcp_client, "calculator", {}, tool_template) + # Streaming + stream = text_protocol.call_tool_streaming(mock_utcp_client, "calculator", {}, tool_template) + chunks = [c async for c in stream] + assert chunks == [content] + finally: + Path(temp_file).unlink() diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index f05222e..0000000 --- a/requirements.txt +++ /dev/null @@ -1,17 +0,0 @@ -pydantic -authlib -python-dotenv -tomli -aiohttp -mcp -gql -pyyaml - -build -pytest -pytest-asyncio -pytest-aiohttp -pytest-cov -coverage -fastapi -uvicorn \ No newline at end of file diff --git a/src/utcp/.DS_Store b/src/utcp/.DS_Store deleted file mode 100644 index 7ff6118..0000000 Binary files a/src/utcp/.DS_Store and /dev/null differ diff --git a/src/utcp/__init__.py b/src/utcp/__init__.py deleted file mode 100644 index 99f872b..0000000 --- a/src/utcp/__init__.py +++ /dev/null @@ -1,42 +0,0 @@ -""" -Universal Tool Calling Protocol Core -""" - -from utcp.shared.tool import ( - Tool, - ToolInputOutputSchema, -) - -from utcp.shared.provider import ( - Provider, - HttpProvider, - CliProvider, - WebSocketProvider, - GRPCProvider, - GraphQLProvider, - TCPProvider, - UDPProvider, - StreamableHttpProvider, - SSEProvider, - WebRTCProvider, - MCPProvider, - TextProvider, -) - -__all__ = [ - "Tool", - "ToolInputOutputSchema", - "Provider", - "HttpProvider", - "CliProvider", - "WebSocketProvider", - "GRPCProvider", - "GraphQLProvider", - "TCPProvider", - "UDPProvider", - "StreamableHttpProvider", - "SSEProvider", - "WebRTCProvider", - "MCPProvider", - "TextProvider", -] diff --git a/src/utcp/client/client_transport_interface.py b/src/utcp/client/client_transport_interface.py deleted file mode 100644 index 1015a8a..0000000 --- a/src/utcp/client/client_transport_interface.py +++ /dev/null @@ -1,84 +0,0 @@ -"""Abstract interface for UTCP client transport implementations. - -This module defines the contract that all transport implementations must follow -to integrate with the UTCP client. Transport implementations handle the actual -communication with different types of tool providers (HTTP, CLI, WebSocket, etc.). -""" - -from abc import ABC, abstractmethod -from typing import Dict, Any, List -from utcp.shared.provider import Provider -from utcp.shared.tool import Tool - -class ClientTransportInterface(ABC): - """Abstract interface for UTCP client transport implementations. - - Defines the contract that all transport implementations must follow to - integrate with the UTCP client. Each transport handles communication - with a specific type of provider (HTTP, CLI, WebSocket, etc.). - - Transport implementations are responsible for: - - Discovering available tools from providers - - Managing provider lifecycle (registration/deregistration) - - Executing tool calls through the appropriate protocol - """ - - @abstractmethod - async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: - """Register a tool provider and discover its available tools. - - Connects to the provider and retrieves the list of tools it offers. - This may involve making discovery requests, parsing configuration files, - or initializing connections depending on the provider type. - - Args: - manual_provider: The provider configuration to register. - - Returns: - List of Tool objects discovered from the provider. - - Raises: - ConnectionError: If unable to connect to the provider. - ValueError: If the provider configuration is invalid. - """ - pass - - @abstractmethod - async def deregister_tool_provider(self, manual_provider: Provider) -> None: - """Deregister a tool provider and clean up resources. - - Cleanly disconnects from the provider and releases any associated - resources such as connections, processes, or file handles. - - Args: - manual_provider: The provider configuration to deregister. - - Note: - Should handle cases where the provider is already disconnected - or was never properly registered. - """ - pass - - @abstractmethod - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> Any: - """Execute a tool call through this transport. - - Sends a tool invocation request to the provider using the appropriate - protocol and returns the result. Handles serialization of arguments - and deserialization of responses according to the transport type. - - Args: - tool_name: Name of the tool to call (may include provider prefix). - arguments: Dictionary of arguments to pass to the tool. - tool_provider: Provider configuration for the tool. - - Returns: - The tool's response, with type depending on the tool's output schema. - - Raises: - ToolNotFoundError: If the specified tool doesn't exist. - ValidationError: If the arguments don't match the tool's input schema. - ConnectionError: If unable to communicate with the provider. - TimeoutError: If the tool call exceeds the configured timeout. - """ - pass diff --git a/src/utcp/client/tool_repositories/in_mem_tool_repository.py b/src/utcp/client/tool_repositories/in_mem_tool_repository.py deleted file mode 100644 index 2ca043b..0000000 --- a/src/utcp/client/tool_repositories/in_mem_tool_repository.py +++ /dev/null @@ -1,48 +0,0 @@ -from typing import List, Dict, Tuple, Optional -from utcp.shared.provider import Provider -from utcp.shared.tool import Tool -from utcp.client.tool_repository import ToolRepository - -class InMemToolRepository(ToolRepository): - def __init__(self): - self.tools: List[Tool] = [] - self.tool_per_provider: Dict[str, Tuple[Provider, List[Tool]]] = {} - - async def save_provider_with_tools(self, provider: Provider, tools: List[Tool]) -> None: - self.tools.extend(tools) - self.tool_per_provider[provider.name] = (provider, tools) - - async def remove_provider(self, provider_name: str) -> None: - if provider_name not in self.tool_per_provider: - raise ValueError(f"Provider '{provider_name}' not found") - tools_to_remove = self.tool_per_provider[provider_name][1] - self.tools = [tool for tool in self.tools if tool not in tools_to_remove] - self.tool_per_provider.pop(provider_name, None) - - async def remove_tool(self, tool_name: str) -> None: - provider_name = tool_name.split(".")[0] - if provider_name not in self.tool_per_provider: - raise ValueError(f"Provider '{provider_name}' not found") - new_tools = [tool for tool in self.tools if tool.name != tool_name] - if len(new_tools) == len(self.tools): - raise ValueError(f"Tool '{tool_name}' not found") - self.tools = new_tools - self.tool_per_provider[provider_name][1] = [tool for tool in self.tool_per_provider[provider_name][1] if tool.name != tool_name] - - async def get_tool(self, tool_name: str) -> Optional[Tool]: - for tool in self.tools: - if tool.name == tool_name: - return tool - return None - - async def get_tools(self) -> List[Tool]: - return self.tools - - async def get_tools_by_provider(self, provider_name: str) -> Optional[List[Tool]]: - return self.tool_per_provider.get(provider_name, (None, None))[1] - - async def get_provider(self, provider_name: str) -> Optional[Provider]: - return self.tool_per_provider.get(provider_name, (None, None))[0] - - async def get_providers(self) -> List[Provider]: - return [provider for provider, _ in self.tool_per_provider.values()] diff --git a/src/utcp/client/tool_repository.py b/src/utcp/client/tool_repository.py deleted file mode 100644 index 3a3278b..0000000 --- a/src/utcp/client/tool_repository.py +++ /dev/null @@ -1,126 +0,0 @@ -"""Abstract interface for tool and provider storage. - -This module defines the contract for implementing tool repositories that store -and manage UTCP tools and their associated providers. Different implementations -can provide various storage backends such as in-memory, database, or file-based -storage. -""" - -from abc import ABC, abstractmethod -from typing import List, Dict, Any, Optional -from utcp.shared.provider import Provider -from utcp.shared.tool import Tool - -class ToolRepository(ABC): - """Abstract interface for tool and provider storage implementations. - - Defines the contract for repositories that manage the lifecycle and storage - of UTCP tools and providers. Repositories are responsible for: - - Persisting provider configurations and their associated tools - - Providing efficient lookup and retrieval operations - - Managing relationships between providers and tools - - Ensuring data consistency during operations - - The repository interface supports both individual and bulk operations, - allowing for flexible implementation strategies ranging from simple - in-memory storage to sophisticated database backends. - - Note: - All methods are async to support both synchronous and asynchronous - storage implementations. - """ - @abstractmethod - async def save_provider_with_tools(self, provider: Provider, tools: List[Tool]) -> None: - """ - Save a provider and its tools in the repository. - - Args: - provider: The provider to save. - tools: The tools associated with the provider. - """ - pass - - @abstractmethod - async def remove_provider(self, provider_name: str) -> None: - """ - Remove a provider and its tools from the repository. - - Args: - provider_name: The name of the provider to remove. - - Raises: - ValueError: If the provider is not found. - """ - pass - - @abstractmethod - async def remove_tool(self, tool_name: str) -> None: - """ - Remove a tool from the repository. - - Args: - tool_name: The name of the tool to remove. - - Raises: - ValueError: If the tool is not found. - """ - pass - - @abstractmethod - async def get_tool(self, tool_name: str) -> Optional[Tool]: - """ - Get a tool from the repository. - - Args: - tool_name: The name of the tool to retrieve. - - Returns: - The tool if found, otherwise None. - """ - pass - - @abstractmethod - async def get_tools(self) -> List[Tool]: - """ - Get all tools from the repository. - - Returns: - A list of tools. - """ - pass - - @abstractmethod - async def get_tools_by_provider(self, provider_name: str) -> Optional[List[Tool]]: - """ - Get tools associated with a specific provider. - - Args: - provider_name: The name of the provider. - - Returns: - A list of tools associated with the provider, or None if the provider is not found. - """ - pass - - @abstractmethod - async def get_provider(self, provider_name: str) -> Optional[Provider]: - """ - Get a provider from the repository. - - Args: - provider_name: The name of the provider to retrieve. - - Returns: - The provider if found, otherwise None. - """ - pass - - @abstractmethod - async def get_providers(self) -> List[Provider]: - """ - Get all providers from the repository. - - Returns: - A list of providers. - """ - pass diff --git a/src/utcp/client/tool_search_strategy.py b/src/utcp/client/tool_search_strategy.py deleted file mode 100644 index b620e34..0000000 --- a/src/utcp/client/tool_search_strategy.py +++ /dev/null @@ -1,48 +0,0 @@ -"""Abstract interface for tool search strategies. - -This module defines the contract for implementing tool search and ranking -algorithms. Different strategies can implement various approaches such as -tag-based search, semantic search, or hybrid approaches. -""" - -from abc import ABC, abstractmethod -from typing import List -from utcp.shared.tool import Tool - -class ToolSearchStrategy(ABC): - """Abstract interface for tool search implementations. - - Defines the contract for tool search strategies that can be plugged into - the UTCP client. Different implementations can provide various search - algorithms such as tag-based matching, semantic similarity, or keyword - search. - - Search strategies are responsible for: - - Interpreting search queries - - Ranking tools by relevance - - Limiting results appropriately - - Providing consistent search behavior - """ - - @abstractmethod - async def search_tools(self, query: str, limit: int = 10) -> List[Tool]: - """Search for tools relevant to the query. - - Executes a search against the available tools and returns the most - relevant matches ranked by the strategy's scoring algorithm. - - Args: - query: The search query string. Format depends on the strategy - (e.g., keywords, tags, natural language). - limit: Maximum number of tools to return. Use 0 for no limit. - Strategies should respect this limit for performance. - - Returns: - List of Tool objects ranked by relevance, limited to the - specified count. Empty list if no matches found. - - Raises: - ValueError: If the query format is invalid for this strategy. - RuntimeError: If the search operation fails unexpectedly. - """ - pass diff --git a/src/utcp/client/transport_interfaces/cli_transport.py b/src/utcp/client/transport_interfaces/cli_transport.py deleted file mode 100644 index 5117465..0000000 --- a/src/utcp/client/transport_interfaces/cli_transport.py +++ /dev/null @@ -1,396 +0,0 @@ -"""Command Line Interface (CLI) transport for UTCP client. - -This module provides the CLI transport implementation that enables UTCP clients -to interact with command-line tools and processes. It handles tool discovery -through startup commands, tool execution with proper argument formatting, -and output processing with JSON parsing capabilities. - -Key Features: - - Asynchronous command execution with timeout handling - - Tool discovery via startup commands that output UTCP manuals - - Flexible argument formatting for command-line flags - - Environment variable support for authentication and configuration - - JSON output parsing with fallback to raw text - - Cross-platform command parsing (Windows/Unix) - - Working directory control for command execution - -Security: - - Command execution is isolated through subprocess - - Environment variables can be controlled per provider - - Working directory can be restricted -""" -import asyncio -import json -import logging -import os -import shlex -import subprocess -from pathlib import Path -from typing import Dict, Any, List, Optional, Callable, Union - -from utcp.client.client_transport_interface import ClientTransportInterface -from utcp.shared.provider import Provider, CliProvider -from utcp.shared.tool import Tool -from utcp.shared.utcp_manual import UtcpManual - - -class CliTransport(ClientTransportInterface): - """Transport implementation for CLI-based tool providers. - - Handles communication with command-line tools by executing processes - and managing their input/output. Supports both tool discovery and - execution phases with comprehensive error handling and timeout management. - - Features: - - Asynchronous subprocess execution with proper cleanup - - Tool discovery through startup commands returning UTCP manuals - - Flexible argument formatting for various CLI conventions - - Environment variable injection for authentication - - JSON output parsing with graceful fallback to text - - Cross-platform command parsing and execution - - Configurable working directories and timeouts - - Process lifecycle management with proper termination - - Architecture: - CLI tools are discovered by executing the provider's command_name - and parsing the output for UTCP manual JSON. Tool calls execute - the same command with formatted arguments and return processed output. - - Attributes: - _log: Logger function for debugging and error reporting. - """ - - def __init__(self, logger: Optional[Callable[[str], None]] = None): - """Initialize the CLI transport. - - Args: - logger: Optional logger function for debugging - """ - self._log = logger or (lambda *args, **kwargs: None) - - def _log_info(self, message: str): - """Log informational messages.""" - self._log(f"[CliTransport] {message}") - - def _log_error(self, message: str): - """Log error messages.""" - logging.error(f"[CliTransport Error] {message}") - - def _prepare_environment(self, provider: CliProvider) -> Dict[str, str]: - """Prepare environment variables for command execution. - - Args: - provider: The CLI provider - - Returns: - Environment variables dictionary - """ - import os - env = os.environ.copy() - - # Add custom environment variables if provided - if provider.env_vars: - env.update(provider.env_vars) - - return env - - async def _execute_command( - self, - command: List[str], - env: Dict[str, str], - timeout: float = 30.0, - input_data: Optional[str] = None, - working_dir: Optional[str] = None - ) -> tuple[str, str, int]: - """Execute a command asynchronously. - - Args: - command: Command and arguments to execute - env: Environment variables - timeout: Timeout in seconds - input_data: Optional input data to pass to the command - working_dir: Working directory for command execution - - Returns: - Tuple of (stdout, stderr, return_code) - """ - process = None - try: - process = await asyncio.create_subprocess_exec( - *command, - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.PIPE, - env=env, - cwd=working_dir, - stdin=asyncio.subprocess.PIPE if input_data else None - ) - - stdout_bytes, stderr_bytes = await asyncio.wait_for( - process.communicate(input=input_data.encode('utf-8') if input_data else None), - timeout=timeout - ) - - stdout = stdout_bytes.decode('utf-8', errors='replace') - stderr = stderr_bytes.decode('utf-8', errors='replace') - - return stdout, stderr, process.returncode or 0 - - except asyncio.TimeoutError: - # Kill the process if it times out - if process: - try: - process.kill() - await process.wait() - except ProcessLookupError: - pass # Process already terminated - self._log_error(f"Command timed out after {timeout} seconds: {' '.join(command)}") - raise - except Exception as e: - # Ensure process is cleaned up on any error - if process: - try: - process.kill() - await process.wait() - except ProcessLookupError: - pass # Process already terminated - self._log_error(f"Error executing command {' '.join(command)}: {e}") - raise - - async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: - """Register a CLI provider and discover its tools. - - Executes the provider's command_name and looks for UTCPManual JSON in the output. - - Args: - manual_provider: The CliProvider to register - - Returns: - List of tools discovered from the CLI provider - - Raises: - ValueError: If provider is not a CliProvider or command_name is not set - """ - if not isinstance(manual_provider, CliProvider): - raise ValueError("CliTransport can only be used with CliProvider") - - if not manual_provider.command_name: - raise ValueError(f"CliProvider '{manual_provider.name}' must have command_name set") - - self._log_info(f"Registering CLI provider '{manual_provider.name}' with command '{manual_provider.command_name}'") - - try: - env = self._prepare_environment(manual_provider) - # Parse command string into proper arguments - # Use posix=False on Windows, posix=True on Unix-like systems - command = shlex.split(manual_provider.command_name, posix=(os.name != 'nt')) - - self._log_info(f"Executing command for tool discovery: {' '.join(command)}") - - stdout, stderr, return_code = await self._execute_command( - command, - env, - timeout=30.0, - working_dir=manual_provider.working_dir - ) - - # Get output based on exit code - output = stdout if return_code == 0 else stderr - - if not output.strip(): - self._log_info(f"No output from command '{manual_provider.command_name}'") - return [] - - # Try to find UTCPManual JSON within the output - tools = self._extract_utcp_manual_from_output(output, manual_provider.name) - - self._log_info(f"Discovered {len(tools)} tools from CLI provider '{manual_provider.name}'") - return tools - - except Exception as e: - self._log_error(f"Error discovering tools from CLI provider '{manual_provider.name}': {e}") - return [] - - async def deregister_tool_provider(self, manual_provider: Provider) -> None: - """Deregister a CLI provider. - - This is a no-op for CLI providers since they are stateless. - - Args: - manual_provider: The provider to deregister - """ - if isinstance(manual_provider, CliProvider): - self._log_info(f"Deregistering CLI provider '{manual_provider.name}' (no-op)") - - def _format_arguments(self, arguments: Dict[str, Any]) -> List[str]: - """Format arguments for command-line execution. - - Converts a dictionary of arguments into command-line flags and values. - - Args: - arguments: Dictionary of argument names and values - - Returns: - List of command-line arguments - """ - args = [] - for key, value in arguments.items(): - if isinstance(value, bool): - if value: - args.append(f"--{key}") - elif isinstance(value, (list, tuple)): - for item in value: - args.extend([f"--{key}", str(item)]) - else: - args.extend([f"--{key}", str(value)]) - return args - - def _extract_utcp_manual_from_output(self, output: str, provider_name: str) -> List[Tool]: - """Extract UTCPManual JSON from command output. - - Searches for JSON content that matches UTCPManual format within the output text. - - Args: - output: The command output to search - provider_name: Name of the provider for logging - - Returns: - List of tools found in the output - """ - tools = [] - - # Try to parse the entire output as JSON first - try: - data = json.loads(output.strip()) - tools = self._parse_tool_data(data, provider_name) - if tools: - return tools - except json.JSONDecodeError: - pass - - # Look for JSON objects within the output text - lines = output.split('\n') - for line in lines: - line = line.strip() - if line.startswith('{') and line.endswith('}'): - try: - data = json.loads(line) - found_tools = self._parse_tool_data(data, provider_name) - tools.extend(found_tools) - except json.JSONDecodeError: - continue - - return tools - - def _parse_tool_data(self, data: Any, provider_name: str) -> List[Tool]: - """Parse tool data from JSON. - - Args: - data: JSON data to parse - provider_name: Name of the provider for logging - - Returns: - List of tools parsed from the data - """ - if isinstance(data, dict): - if 'tools' in data: - # Standard UTCP manual format - try: - utcp_manual = UtcpManual(**data) - return utcp_manual.tools - except Exception as e: - self._log_error(f"Invalid UTCP manual format from provider '{provider_name}': {e}") - return [] - elif 'name' in data and 'description' in data: - # Single tool definition - try: - return [Tool(**data)] - except Exception as e: - self._log_error(f"Invalid tool definition from provider '{provider_name}': {e}") - return [] - elif isinstance(data, list): - # Array of tool definitions - try: - return [Tool(**tool_data) for tool_data in data] - except Exception as e: - self._log_error(f"Invalid tool array from provider '{provider_name}': {e}") - return [] - - return [] - - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> Any: - """Call a CLI tool. - - Executes the command specified by provider.command_name with the provided arguments. - - Args: - tool_name: Name of the tool to call - arguments: Arguments for the tool call - tool_provider: The CliProvider containing the tool - - Returns: - The output from the command execution based on exit code: - - If exit code is 0: stdout (parsed as JSON if possible, otherwise raw string) - - If exit code is not 0: stderr - - Raises: - ValueError: If provider is not a CliProvider or command_name is not set - """ - if not isinstance(tool_provider, CliProvider): - raise ValueError("CliTransport can only be used with CliProvider") - - if not tool_provider.command_name: - raise ValueError(f"CliProvider '{tool_provider.name}' must have command_name set") - - # Build the command - # Parse command string into proper arguments - # Use posix=False on Windows, posix=True on Unix-like systems - command = shlex.split(tool_provider.command_name, posix=(os.name != 'nt')) - - # Add formatted arguments - if arguments: - command.extend(self._format_arguments(arguments)) - - self._log_info(f"Executing CLI tool '{tool_name}': {' '.join(command)}") - - try: - env = self._prepare_environment(tool_provider) - - stdout, stderr, return_code = await self._execute_command( - command, - env, - timeout=60.0, # Longer timeout for tool execution - working_dir=tool_provider.working_dir - ) - - # Get output based on exit code - if return_code == 0: - output = stdout - self._log_info(f"CLI tool '{tool_name}' executed successfully (exit code 0)") - else: - output = stderr - self._log_info(f"CLI tool '{tool_name}' exited with code {return_code}, returning stderr") - - # Try to parse output as JSON, fall back to raw string - if output.strip(): - try: - result = json.loads(output) - self._log_info(f"Returning JSON output from CLI tool '{tool_name}'") - return result - except json.JSONDecodeError: - # Return raw string output - self._log_info(f"Returning text output from CLI tool '{tool_name}'") - return output.strip() - else: - self._log_info(f"CLI tool '{tool_name}' produced no output") - return "" - - except Exception as e: - self._log_error(f"Error executing CLI tool '{tool_name}': {e}") - raise - - async def close(self) -> None: - """Close the transport. - - This is a no-op for CLI transports since they don't maintain connections. - """ - self._log_info("Closing CLI transport (no-op)") diff --git a/src/utcp/client/transport_interfaces/text_transport.py b/src/utcp/client/transport_interfaces/text_transport.py deleted file mode 100644 index daf6faa..0000000 --- a/src/utcp/client/transport_interfaces/text_transport.py +++ /dev/null @@ -1,166 +0,0 @@ -""" -Text file transport for UTCP client. - -This transport reads tool definitions from local text files. -""" -import json -import logging -import yaml -from pathlib import Path -from typing import Dict, Any, List, Optional, Callable - -from utcp.client.client_transport_interface import ClientTransportInterface -from utcp.client.openapi_converter import OpenApiConverter -from utcp.shared.provider import Provider, TextProvider -from utcp.shared.tool import Tool -from utcp.shared.utcp_manual import UtcpManual - - -class TextTransport(ClientTransportInterface): - """Transport implementation for text file-based tool providers. - - This transport reads tool definitions from local text files. The file should - contain a JSON object with a 'tools' array containing tool definitions. - - Since tools are defined statically in text files, tool calls are not supported - and will raise a ValueError. - """ - - def __init__(self, base_path: Optional[str] = None): - """Initialize the text transport. - - Args: - base_path: The base path to resolve relative file paths from. - """ - self.base_path = base_path - - def _log_info(self, message: str): - """Log informational messages.""" - print(f"[TextTransport] {message}") - - def _log_error(self, message: str): - """Log error messages.""" - logging.error(f"[TextTransport Error] {message}") - - async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: - """Register a text provider and discover its tools. - - Args: - manual_provider: The TextProvider to register - - Returns: - List of tools defined in the text file - - Raises: - ValueError: If provider is not a TextProvider - FileNotFoundError: If the specified file doesn't exist - json.JSONDecodeError: If the file contains invalid JSON - """ - if not isinstance(manual_provider, TextProvider): - raise ValueError("TextTransport can only be used with TextProvider") - - file_path = Path(manual_provider.file_path) - if not file_path.is_absolute() and self.base_path: - file_path = Path(self.base_path) / file_path - - self._log_info(f"Reading tool definitions from '{file_path}'") - - try: - if not file_path.exists(): - raise FileNotFoundError(f"Tool definition file not found: {file_path}") - - with open(file_path, 'r', encoding='utf-8') as f: - file_content = f.read() - - # Parse based on file extension - if file_path.suffix in ['.yaml', '.yml']: - data = yaml.safe_load(file_content) - else: - data = json.loads(file_content) - - # Check if the data is a UTCP manual, an OpenAPI spec, or neither - if isinstance(data, dict) and "version" in data and "tools" in data: - self._log_info(f"Detected UTCP manual in '{file_path}'.") - utcp_manual = UtcpManual(**data) - elif isinstance(data, dict) and ('openapi' in data or 'swagger' in data or 'paths' in data): - self._log_info(f"Assuming OpenAPI spec in '{file_path}'. Converting to UTCP manual.") - converter = OpenApiConverter(data, spec_url=file_path.as_uri(), provider_name=manual_provider.name) - utcp_manual = converter.convert() - else: - raise ValueError(f"File '{file_path}' is not a valid OpenAPI specification or UTCP manual") - - self._log_info(f"Successfully loaded {len(utcp_manual.tools)} tools from '{file_path}'") - return utcp_manual.tools - - except FileNotFoundError: - self._log_error(f"Tool definition file not found: {file_path}") - raise - except (json.JSONDecodeError, yaml.YAMLError) as e: - self._log_error(f"Failed to parse file '{file_path}': {e}") - raise - except Exception as e: - self._log_error(f"Unexpected error reading file '{file_path}': {e}") - return [] - - async def deregister_tool_provider(self, manual_provider: Provider) -> None: - """Deregister a text provider. - - This is a no-op for text providers since they are stateless. - - Args: - manual_provider: The provider to deregister - """ - if isinstance(manual_provider, TextProvider): - self._log_info(f"Deregistering text provider '{manual_provider.name}' (no-op)") - - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> Any: - """Call a tool on a text provider. - - For text providers, this returns the content of the text file. - - Args: - tool_name: Name of the tool to call (ignored for text providers) - arguments: Arguments for the tool call (ignored for text providers) - provider: The TextProvider containing the file - - Returns: - The content of the text file as a string - - Raises: - ValueError: If provider is not a TextProvider - FileNotFoundError: If the specified file doesn't exist - """ - if not isinstance(tool_provider, TextProvider): - raise ValueError("TextTransport can only be used with TextProvider") - - file_path = Path(tool_provider.file_path) - if not file_path.is_absolute() and self.base_path: - file_path = Path(self.base_path) / file_path - - self._log_info(f"Reading content from '{file_path}' for tool '{tool_name}'") - - try: - # Check if file exists - if not file_path.exists(): - raise FileNotFoundError(f"File not found: {file_path}") - - # Read and return the file content - with open(file_path, 'r', encoding='utf-8') as f: - content = f.read() - - self._log_info(f"Successfully read {len(content)} characters from '{file_path}'") - return content - - except FileNotFoundError: - self._log_error(f"File not found: {file_path}") - raise - except Exception as e: - self._log_error(f"Error reading file '{file_path}': {e}") - raise - - async def close(self) -> None: - """Close the transport. - - This is a no-op for text transports since they don't maintain connections. - """ - self._log_info("Closing text transport (no-op)") diff --git a/src/utcp/client/utcp_client.py b/src/utcp/client/utcp_client.py deleted file mode 100644 index f3505c0..0000000 --- a/src/utcp/client/utcp_client.py +++ /dev/null @@ -1,435 +0,0 @@ -"""Main UTCP client implementation. - -This module provides the primary client interface for the Universal Tool Calling -Protocol. The UtcpClient class manages multiple transport implementations, -tool repositories, search strategies, and provider configurations. - -Key Features: - - Multi-transport support (HTTP, CLI, WebSocket, etc.) - - Dynamic provider registration and deregistration - - Tool discovery and search capabilities - - Variable substitution for configuration - - Pluggable tool repositories and search strategies -""" - -from pathlib import Path -import re -import os -import json -import asyncio -from abc import ABC, abstractmethod -from typing import Dict, Any, List, Union, Optional -from utcp.shared.tool import Tool -from utcp.client.client_transport_interface import ClientTransportInterface -from utcp.client.transport_interfaces.http_transport import HttpClientTransport -from utcp.client.transport_interfaces.cli_transport import CliTransport -from utcp.client.transport_interfaces.sse_transport import SSEClientTransport -from utcp.client.transport_interfaces.streamable_http_transport import StreamableHttpClientTransport -from utcp.client.transport_interfaces.mcp_transport import MCPTransport -from utcp.client.transport_interfaces.text_transport import TextTransport -from utcp.client.transport_interfaces.graphql_transport import GraphQLClientTransport -from utcp.client.transport_interfaces.tcp_transport import TCPTransport -from utcp.client.transport_interfaces.udp_transport import UDPTransport -from utcp.client.utcp_client_config import UtcpClientConfig, UtcpVariableNotFound -from utcp.client.tool_repository import ToolRepository -from utcp.client.tool_repositories.in_mem_tool_repository import InMemToolRepository -from utcp.client.tool_search_strategies.tag_search import TagSearchStrategy -from utcp.client.tool_search_strategy import ToolSearchStrategy -from utcp.shared.provider import Provider, HttpProvider, CliProvider, SSEProvider, \ - StreamableHttpProvider, WebSocketProvider, GRPCProvider, GraphQLProvider, \ - TCPProvider, UDPProvider, WebRTCProvider, MCPProvider, TextProvider -from utcp.client.variable_substitutor import DefaultVariableSubstitutor, VariableSubstitutor - -class UtcpClientInterface(ABC): - """Abstract interface for UTCP client implementations. - - Defines the core contract for UTCP clients, including provider management, - tool execution, search capabilities, and variable handling. This interface - allows for different client implementations while maintaining consistency. - - The interface supports: - - Provider lifecycle management (register/deregister) - - Tool discovery and execution - - Tool search and filtering - - Configuration variable validation - """ - @abstractmethod - def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: - """ - Register a tool provider and its tools. - - Args: - manual_provider: The provider to register. - - Returns: - A list of tools associated with the provider. - """ - pass - - @abstractmethod - def deregister_tool_provider(self, provider_name: str) -> None: - """ - Deregister a tool provider. - - Args: - provider_name: The name of the provider to deregister. - """ - pass - - @abstractmethod - def call_tool(self, tool_name: str, arguments: Dict[str, Any]) -> Any: - """ - Call a tool. - - Args: - tool_name: The name of the tool to call. - arguments: The arguments to pass to the tool. - - Returns: - The result of the tool call. - """ - pass - - @abstractmethod - def search_tools(self, query: str, limit: int = 10) -> List[Tool]: - """ - Search for tools relevant to the query. - - Args: - query: The search query. - limit: The maximum number of tools to return. 0 for no limit. - - Returns: - A list of tools that match the search query. - """ - pass - - @abstractmethod - def get_required_variables_for_manual_and_tools(self, manual_provider: Provider) -> List[str]: - """ - Get the required variables for a manual provider and its tools. - - Args: - manual_provider: The manual provider. - - Returns: - A list of required variables for the manual provider and its tools. - """ - pass - - @abstractmethod - def get_required_variables_for_tool(self, tool_name: str) -> List[str]: - """ - Get the required variables for a registered tool. - - Args: - tool_name: The name of a registered tool. - - Returns: - A list of required variables for the tool. - """ - pass - -class UtcpClient(UtcpClientInterface): - """Main implementation of the UTCP client. - - The UtcpClient is the primary entry point for interacting with UTCP tool - providers. It manages multiple transport implementations, handles provider - registration, executes tool calls, and provides search capabilities. - - Key Features: - - Multi-transport architecture supporting HTTP, CLI, WebSocket, etc. - - Dynamic provider registration from configuration files - - Variable substitution for secure credential management - - Pluggable tool repositories and search strategies - - Comprehensive error handling and validation - - Architecture: - - Transport Layer: Handles protocol-specific communication - - Repository Layer: Manages tool and provider storage - - Search Layer: Provides tool discovery and filtering - - Configuration Layer: Manages settings and variable substitution - - Usage: - >>> client = await UtcpClient.create({ - ... "providers_file_path": "./providers.json" - ... }) - >>> tools = await client.search_tools("weather") - >>> result = await client.call_tool("api.get_weather", {"city": "NYC"}) - - Attributes: - transports: Dictionary mapping provider types to transport implementations. - tool_repository: Storage backend for tools and providers. - search_strategy: Algorithm for tool search and ranking. - config: Client configuration including file paths and settings. - variable_substitutor: Handler for environment variable substitution. - """ - - transports: Dict[str, ClientTransportInterface] = { - "http": HttpClientTransport(), - "cli": CliTransport(), - "sse": SSEClientTransport(), - "http_stream": StreamableHttpClientTransport(), - "mcp": MCPTransport(), - "text": TextTransport(), - "graphql": GraphQLClientTransport(), - "tcp": TCPTransport(), - "udp": UDPTransport(), - } - - def __init__(self, config: UtcpClientConfig, tool_repository: ToolRepository, search_strategy: ToolSearchStrategy, variable_substitutor: VariableSubstitutor): - """ - Use 'create' class method to create a new instance instead, as it supports loading UtcpClientConfig. - """ - self.tool_repository = tool_repository - self.search_strategy = search_strategy - self.config = config - self.variable_substitutor = variable_substitutor - - @classmethod - async def create(cls, config: Optional[Union[Dict[str, Any], UtcpClientConfig]] = None, tool_repository: Optional[ToolRepository] = None, search_strategy: Optional[ToolSearchStrategy] = None) -> 'UtcpClient': - """ - Create a new instance of UtcpClient. - - Args: - config: The configuration for the client. Can be a dictionary or UtcpClientConfig object. - tool_repository: The tool repository to use. Defaults to InMemToolRepository. - search_strategy: The tool search strategy to use. Defaults to TagSearchStrategy. - - Returns: - A new instance of UtcpClient. - """ - if tool_repository is None: - tool_repository = InMemToolRepository() - if search_strategy is None: - search_strategy = TagSearchStrategy(tool_repository) - if config is None: - config = UtcpClientConfig() - elif isinstance(config, dict): - config = UtcpClientConfig.model_validate(config) - - client = cls(config, tool_repository, search_strategy, DefaultVariableSubstitutor()) - - if client.config.variables: - config_without_vars = client.config.model_copy() - config_without_vars.variables = None - client.config.variables = client.variable_substitutor.substitute(client.config.variables, config_without_vars) - - # If a providers file is used, configure TextTransport to resolve relative paths from its directory - if config.providers_file_path: - providers_dir = os.path.dirname(os.path.abspath(config.providers_file_path)) - client.transports["text"] = TextTransport(base_path=providers_dir) - - await client.load_providers(config.providers_file_path) - - return client - - async def load_providers(self, providers_file_path: str) -> List[Provider]: - """Load providers from the file specified in the configuration. - - Returns: - List of registered Provider objects. - - Raises: - FileNotFoundError: If the providers file doesn't exist. - ValueError: If the providers file contains invalid JSON. - UtcpVariableNotFound: If a variable referenced in the provider configuration is not found. - """ - if not providers_file_path: - return [] - - providers_file_path = Path(providers_file_path).resolve() - try: - with open(providers_file_path, 'r') as f: - providers_data = json.load(f) - except FileNotFoundError: - raise FileNotFoundError(f"Providers file not found: {providers_file_path}") - except json.JSONDecodeError: - raise ValueError(f"Invalid JSON in providers file: {providers_file_path}") - - provider_classes = { - 'http': HttpProvider, - 'cli': CliProvider, - 'sse': SSEProvider, - 'http_stream': StreamableHttpProvider, - 'websocket': WebSocketProvider, - 'grpc': GRPCProvider, - 'graphql': GraphQLProvider, - 'tcp': TCPProvider, - 'udp': UDPProvider, - 'webrtc': WebRTCProvider, - 'mcp': MCPProvider, - 'text': TextProvider - } - - if not isinstance(providers_data, list): - raise ValueError(f"Providers file must contain a JSON array at the root level: {providers_file_path}") - - registered_providers = [] - # Create tasks for parallel provider registration - tasks = [] - for provider_data in providers_data: - async def register_single_provider(provider_data=provider_data): - try: - # Determine provider type from provider_type field - provider_type = provider_data.get('provider_type') - if not provider_type: - print(f"Warning: Provider entry is missing required 'provider_type' field, skipping: {provider_data}") - return None - - provider_class = provider_classes.get(provider_type) - if not provider_class: - print(f"Warning: Unsupported provider type: {provider_type}, skipping") - return None - - # Create provider object with Pydantic validation - provider = provider_class.model_validate(provider_data) - - # Apply variable substitution and register provider - tools = await self.register_tool_provider(provider) - print(f"Successfully registered provider '{provider.name}' with {len(tools)} tools") - return provider - except Exception as e: - # Log the error but continue with other providers - provider_name = provider_data.get('name', 'unknown') - print(f"Error registering provider '{provider_name}': {str(e)}") - return None - - tasks.append(register_single_provider()) - - # Wait for all tasks to complete and collect results - results = await asyncio.gather(*tasks) - registered_providers = [p for p in results if p is not None] - - return registered_providers - - def _substitute_provider_variables(self, provider: Provider, provider_name: Optional[str] = None) -> Provider: - provider_dict = provider.model_dump() - - processed_dict = self.variable_substitutor.substitute(provider_dict, self.config, provider_name) - return provider.__class__(**processed_dict) - - async def get_required_variables_for_manual_and_tools(self, manual_provider: Provider) -> List[str]: - """ - Get the required variables for a manual provider and its tools. - - Args: - manual_provider: The provider to validate. - - Returns: - A list of required variables for the provider. - - Raises: - ValueError: If the provider type is not supported. - UtcpVariableNotFound: If a variable is not found in the environment or in the configuration. - """ - manual_provider.name = re.sub(r'[^\w]', '_', manual_provider.name) - variables_for_provider = self.variable_substitutor.find_required_variables(manual_provider.model_dump(), manual_provider.name) - if len(variables_for_provider) > 0: - try: - manual_provider = self._substitute_provider_variables(manual_provider, manual_provider.name) - except UtcpVariableNotFound as e: - return variables_for_provider - return variables_for_provider - if manual_provider.provider_type not in self.transports: - raise ValueError(f"Provider type not supported: {manual_provider.provider_type}") - tools: List[Tool] = await self.transports[manual_provider.provider_type].register_tool_provider(manual_provider) - for tool in tools: - variables_for_provider.extend(self.variable_substitutor.find_required_variables(tool.tool_provider.model_dump(), manual_provider.name)) - return variables_for_provider - - async def get_required_variables_for_tool(self, tool_name: str) -> List[str]: - """ - Get the required variables for a tool. - - Args: - tool_name: The name of the tool to validate. - - Returns: - A list of required variables for the tool. - - Raises: - ValueError: If the provider type is not supported. - UtcpVariableNotFound: If a variable is not found in the environment or in the configuration. - """ - provider_name = tool_name.split(".")[0] - tool = await self.tool_repository.get_tool(tool_name) - if tool is None: - raise ValueError(f"Tool not found: {tool_name}") - return self.variable_substitutor.find_required_variables(tool.tool_provider.model_dump(), provider_name) - - async def register_tool_provider(self, manual_provider: Provider) -> List[Tool]: - """ - Register a tool provider. - - Args: - manual_provider: The provider to register. - - Returns: - A list of tools registered by the provider. - - Raises: - ValueError: If the provider type is not supported. - UtcpVariableNotFound: If a variable is not found in the environment or in the configuration. - """ - # Replace all non-word characters with underscore - manual_provider.name = re.sub(r'[^\w]', '_', manual_provider.name) - if await self.tool_repository.get_provider(manual_provider.name) is not None: - raise ValueError(f"Provider {manual_provider.name} already registered, please use a different name or deregister the existing provider") - manual_provider = self._substitute_provider_variables(manual_provider, manual_provider.name) - if manual_provider.provider_type not in self.transports: - raise ValueError(f"Provider type not supported: {manual_provider.provider_type}") - tools: List[Tool] = await self.transports[manual_provider.provider_type].register_tool_provider(manual_provider) - for tool in tools: - if not tool.name.startswith(manual_provider.name + "."): - tool.name = manual_provider.name + "." + tool.name - await self.tool_repository.save_provider_with_tools(manual_provider, tools) - return tools - - async def deregister_tool_provider(self, provider_name: str) -> None: - """ - Deregister a tool provider. - - Args: - provider_name: The name of the provider to deregister. - - Raises: - ValueError: If the provider is not found. - """ - provider = await self.tool_repository.get_provider(provider_name) - if provider is None: - raise ValueError(f"Provider not found: {provider_name}") - await self.transports[provider.provider_type].deregister_tool_provider(provider) - await self.tool_repository.remove_provider(provider_name) - - async def call_tool(self, tool_name: str, arguments: Dict[str, Any]) -> Any: - """ - Call a tool. - - Args: - tool_name: The name of the tool to call. Should be in the format provider_name.tool_name. - arguments: The arguments to pass to the tool. - - Returns: - The result of the tool. - - Raises: - ValueError: If the tool is not found. - UtcpVariableNotFound: If a variable is not found in the environment or in the configuration. - """ - manual_provider_name = tool_name.split(".")[0] - manual_provider = await self.tool_repository.get_provider(manual_provider_name) - if manual_provider is None: - raise ValueError(f"Provider not found: {manual_provider_name}") - tool = await self.tool_repository.get_tool(tool_name) - if tool is None: - raise ValueError(f"Tool not found: {tool_name}") - - tool_provider = tool.tool_provider - - tool_provider = self._substitute_provider_variables(tool_provider, manual_provider_name) - - return await self.transports[tool_provider.provider_type].call_tool(tool_name, arguments, tool_provider) - - async def search_tools(self, query: str, limit: int = 10) -> List[Tool]: - return await self.search_strategy.search_tools(query, limit) diff --git a/src/utcp/client/utcp_client_config.py b/src/utcp/client/utcp_client_config.py deleted file mode 100644 index 0ffb2fa..0000000 --- a/src/utcp/client/utcp_client_config.py +++ /dev/null @@ -1,139 +0,0 @@ -"""Configuration models for UTCP client setup. - -This module defines the configuration classes and variable loading mechanisms -for UTCP clients. It provides flexible variable substitution support through -multiple sources including environment files, direct configuration, and -custom variable loaders. - -The configuration system enables: - - Variable substitution in provider configurations - - Multiple variable sources with hierarchical resolution - - Environment file loading (.env files) - - Direct variable specification - - Custom variable loader implementations -""" - -from abc import ABC, abstractmethod -from pydantic import BaseModel, Field -from typing import Optional, List, Dict, Annotated, Union, Literal -from dotenv import dotenv_values - -class UtcpVariableNotFound(Exception): - """Exception raised when a required variable cannot be found. - - This exception is thrown during variable substitution when a referenced - variable cannot be resolved through any of the configured variable sources. - It provides information about which variable was missing to help with - debugging configuration issues. - - Attributes: - variable_name: The name of the variable that could not be found. - """ - variable_name: str - - def __init__(self, variable_name: str): - """Initialize the exception with the missing variable name. - - Args: - variable_name: Name of the variable that could not be found. - """ - self.variable_name = variable_name - super().__init__(f"Variable {variable_name} referenced in provider configuration not found. Please add it to the environment variables or to your UTCP configuration.") - -class UtcpVariablesConfig(BaseModel, ABC): - """Abstract base class for variable loading configurations. - - Defines the interface for variable loaders that can retrieve variable - values from different sources such as files, databases, or external - services. Implementations provide specific loading mechanisms while - maintaining a consistent interface. - - Attributes: - type: Type identifier for the variable loader. - """ - type: str - - @abstractmethod - def get(self, key: str) -> Optional[str]: - """Retrieve a variable value by key. - - Args: - key: Variable name to retrieve. - - Returns: - Variable value if found, None otherwise. - """ - pass - -class UtcpDotEnv(UtcpVariablesConfig): - """Environment file variable loader implementation. - - Loads variables from .env files using the dotenv format. This loader - supports the standard key=value format with optional quoting and - comment support provided by the python-dotenv library. - - Attributes: - env_file_path: Path to the .env file to load variables from. - - Example: - ```python - loader = UtcpDotEnv(env_file_path=".env") - api_key = loader.get("API_KEY") - ``` - """ - type: Literal["dotenv"] = "dotenv" - env_file_path: str - - def get(self, key: str) -> Optional[str]: - """Load a variable from the configured .env file. - - Args: - key: Variable name to retrieve from the environment file. - - Returns: - Variable value if found in the file, None otherwise. - """ - return dotenv_values(self.env_file_path).get(key) - -UtcpVariablesConfigUnion = Annotated[ - Union[ - UtcpDotEnv - ], - Field(discriminator="type") -] - -class UtcpClientConfig(BaseModel): - """Configuration model for UTCP client setup. - - Provides comprehensive configuration options for UTCP clients including - variable definitions, provider file locations, and variable loading - mechanisms. Supports hierarchical variable resolution with multiple - sources. - - Variable Resolution Order: - 1. Direct variables dictionary - 2. Custom variable loaders (in order) - 3. Environment variables - - Attributes: - variables: Direct variable definitions as key-value pairs. - These take precedence over other variable sources. - providers_file_path: Optional path to a file containing provider - configurations. Supports JSON and YAML formats. - load_variables_from: List of variable loaders to use for - variable resolution. Loaders are consulted in order. - - Example: - ```python - config = UtcpClientConfig( - variables={"API_BASE": "https://api.example.com"}, - providers_file_path="providers.yaml", - load_variables_from=[ - UtcpDotEnv(env_file_path=".env") - ] - ) - ``` - """ - variables: Optional[Dict[str, str]] = Field(default_factory=dict) - providers_file_path: Optional[str] = None - load_variables_from: Optional[List[UtcpVariablesConfigUnion]] = None diff --git a/src/utcp/shared/auth.py b/src/utcp/shared/auth.py deleted file mode 100644 index b06f2b9..0000000 --- a/src/utcp/shared/auth.py +++ /dev/null @@ -1,79 +0,0 @@ -"""Authentication schemes for UTCP providers. - -This module defines the authentication models supported by UTCP providers, -including API key authentication, basic authentication, and OAuth2. -""" - -from typing import Literal, Optional, TypeAlias, Union - -from pydantic import BaseModel, Field - -class ApiKeyAuth(BaseModel): - """Authentication using an API key. - - The key can be provided directly or sourced from an environment variable. - Supports placement in headers, query parameters, or cookies. - - Attributes: - auth_type: The authentication type identifier, always "api_key". - api_key: The API key for authentication. Values starting with '$' or formatted as '${}' are - treated as an injected variable from environment or configuration. - var_name: The name of the header, query parameter, or cookie that - contains the API key. - location: Where to include the API key (header, query parameter, or cookie). - """ - - auth_type: Literal["api_key"] = "api_key" - api_key: str = Field(..., description="The API key for authentication. Values starting with '$' or formatted as '${}' are treated as an injected variable from environment or configuration. This is the recommended way to provide API keys.") - var_name: str = Field( - "X-Api-Key", description="The name of the header, query parameter, cookie or other container for the API key." - ) - location: Literal["header", "query", "cookie"] = Field( - "header", description="Where to include the API key (header, query parameter, or cookie)." - ) - - -class BasicAuth(BaseModel): - """Authentication using HTTP Basic Authentication. - - Uses the standard HTTP Basic Authentication scheme with username and password - encoded in the Authorization header. - - Attributes: - auth_type: The authentication type identifier, always "basic". - username: The username for basic authentication. Recommended to use injected variables. - password: The password for basic authentication. Recommended to use injected variables. - """ - - auth_type: Literal["basic"] = "basic" - username: str = Field(..., description="The username for basic authentication.") - password: str = Field(..., description="The password for basic authentication.") - - -class OAuth2Auth(BaseModel): - """Authentication using OAuth2 client credentials flow. - - Implements the OAuth2 client credentials grant type for machine-to-machine - authentication. The client automatically handles token acquisition and refresh. - - Attributes: - auth_type: The authentication type identifier, always "oauth2". - token_url: The URL endpoint to fetch the OAuth2 access token from. Recommended to use injected variables. - client_id: The OAuth2 client identifier. Recommended to use injected variables. - client_secret: The OAuth2 client secret. Recommended to use injected variables. - scope: Optional scope parameter to limit the access token's permissions. - """ - - auth_type: Literal["oauth2"] = "oauth2" - token_url: str = Field(..., description="The URL to fetch the OAuth2 token from.") - client_id: str = Field(..., description="The OAuth2 client ID.") - client_secret: str = Field(..., description="The OAuth2 client secret.") - scope: Optional[str] = Field(None, description="The OAuth2 scope.") - - -Auth: TypeAlias = Union[ApiKeyAuth, BasicAuth, OAuth2Auth] -"""Type alias for all supported authentication schemes. - -This union type encompasses all authentication methods supported by UTCP providers. -Use this type for type hints when accepting any authentication scheme. -""" diff --git a/src/utcp/shared/provider.py b/src/utcp/shared/provider.py deleted file mode 100644 index 99e50dc..0000000 --- a/src/utcp/shared/provider.py +++ /dev/null @@ -1,513 +0,0 @@ -"""Provider configurations for UTCP tool providers. - -This module defines the provider models and configurations for all supported -transport protocols in UTCP. Each provider type encapsulates the necessary -configuration to connect to and interact with tools through different -communication channels. - -Supported provider types: - - HTTP: RESTful HTTP/HTTPS APIs - - SSE: Server-Sent Events for streaming - - HTTP Stream: HTTP Chunked Transfer Encoding - - CLI: Command Line Interface tools - - WebSocket: Bidirectional WebSocket connections (WIP) - - gRPC: Google Remote Procedure Call (WIP) - - GraphQL: GraphQL query language - - TCP: Raw TCP socket connections - - UDP: User Datagram Protocol - - WebRTC: Web Real-Time Communication (WIP) - - MCP: Model Context Protocol - - Text: Text file-based providers -""" - -from typing import Dict, Any, Optional, List, Literal, TypeAlias, Union -from pydantic import BaseModel, Field -from typing import Annotated -import uuid -from utcp.shared.auth import ( - Auth, - ApiKeyAuth, - BasicAuth, - OAuth2Auth, -) - -ProviderType: TypeAlias = Literal[ - 'http', # RESTful HTTP/HTTPS API - 'sse', # Server-Sent Events - 'http_stream', # HTTP Chunked Transfer Encoding - 'cli', # Command Line Interface - 'websocket', # WebSocket bidirectional connection (WIP) - 'grpc', # gRPC (Google Remote Procedure Call) (WIP) - 'graphql', # GraphQL query language - 'tcp', # Raw TCP socket - 'udp', # User Datagram Protocol - 'webrtc', # Web Real-Time Communication (WIP) - 'mcp', # Model Context Protocol - 'text', # Text file provider -] -"""Type alias for all supported provider transport types. - -This literal type defines all the communication protocols and transport -mechanisms that UTCP supports for connecting to tool providers. -""" - -class Provider(BaseModel): - """Base class for all UTCP tool providers. - - This is the abstract base class that all specific provider implementations - inherit from. It provides the common fields that every provider must have. - - Attributes: - name: Unique identifier for the provider. Defaults to a random UUID hex string. - Should be unique across all providers and recommended to be set to a human-readable name. - Can only contain letters, numbers and underscores. All special characters must be replaced with underscores. - provider_type: The transport protocol type used by this provider. - """ - - name: str = uuid.uuid4().hex - provider_type: ProviderType - -class HttpProvider(Provider): - """Provider configuration for HTTP-based tools. - - Supports RESTful HTTP/HTTPS APIs with various HTTP methods, authentication, - custom headers, and flexible request/response handling. Supports URL path - parameters using {parameter_name} syntax. All tool arguments not mapped to - URL body, headers or query pattern parameters are passed as query parameters using '?arg_name={arg_value}'. - - Attributes: - provider_type: Always "http" for HTTP providers. - http_method: The HTTP method to use for requests. - url: The base URL for the HTTP endpoint. Supports path parameters like - "https://api.example.com/users/{user_id}/posts/{post_id}". - content_type: The Content-Type header for requests. - auth: Optional authentication configuration. - headers: Optional static headers to include in all requests. - body_field: Name of the tool argument to map to the HTTP request body. - header_fields: List of tool argument names to map to HTTP request headers. - """ - - provider_type: Literal["http"] = "http" - http_method: Literal["GET", "POST", "PUT", "DELETE", "PATCH"] = "GET" - url: str - content_type: str = Field(default="application/json") - auth: Optional[Auth] = None - headers: Optional[Dict[str, str]] = None - body_field: Optional[str] = Field(default="body", description="The name of the single input field to be sent as the request body.") - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers.") - -class SSEProvider(Provider): - """Provider configuration for Server-Sent Events (SSE) tools. - - Enables real-time streaming of events from server to client using the - Server-Sent Events protocol. Supports automatic reconnection and - event type filtering. All tool arguments not mapped to URL body, headers - or query pattern parameters are passed as query parameters using '?arg_name={arg_value}'. - - Attributes: - provider_type: Always "sse" for SSE providers. - url: The SSE endpoint URL to connect to. - event_type: Optional filter for specific event types. If None, all events are received. - reconnect: Whether to automatically reconnect on connection loss. - retry_timeout: Timeout in milliseconds before attempting reconnection. - auth: Optional authentication configuration. - headers: Optional static headers for the initial connection. - body_field: Optional tool argument name to map to request body during connection. - header_fields: List of tool argument names to map to HTTP headers during connection. - """ - - provider_type: Literal["sse"] = "sse" - url: str - event_type: Optional[str] = None - reconnect: bool = True - retry_timeout: int = 30000 # Retry timeout in milliseconds if disconnected - auth: Optional[Auth] = None - headers: Optional[Dict[str, str]] = None - body_field: Optional[str] = Field(default=None, description="The name of the single input field to be sent as the request body.") - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers for the initial connection.") - -class StreamableHttpProvider(Provider): - """Provider configuration for HTTP streaming tools. - - Uses HTTP Chunked Transfer Encoding to enable streaming of large responses - or real-time data. Useful for tools that return large datasets or provide - progressive results. All tool arguments not mapped to URL body, headers - or query pattern parameters are passed as query parameters using '?arg_name={arg_value}'. - - Attributes: - provider_type: Always "http_stream" for HTTP streaming providers. - url: The streaming HTTP endpoint URL. Supports path parameters. - http_method: The HTTP method to use (GET or POST). - content_type: The Content-Type header for requests. - chunk_size: Size of each chunk in bytes for reading the stream. - timeout: Request timeout in milliseconds. - headers: Optional static headers to include in requests. - auth: Optional authentication configuration. - body_field: Optional tool argument name to map to HTTP request body. - header_fields: List of tool argument names to map to HTTP request headers. - """ - - provider_type: Literal["http_stream"] = "http_stream" - url: str - http_method: Literal["GET", "POST"] = "GET" - content_type: str = "application/octet-stream" - chunk_size: int = 4096 # Size of chunks in bytes - timeout: int = 60000 # Timeout in milliseconds - headers: Optional[Dict[str, str]] = None - auth: Optional[Auth] = None - body_field: Optional[str] = Field(default=None, description="The name of the single input field to be sent as the request body.") - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers.") - -class CliProvider(Provider): - """Provider configuration for Command Line Interface tools. - - Enables execution of command-line tools and programs as UTCP providers. - Supports environment variable injection and custom working directories. - - Attributes: - provider_type: Always "cli" for CLI providers. - command_name: The name or path of the command to execute. - env_vars: Optional environment variables to set during command execution. - working_dir: Optional custom working directory for command execution. - auth: Always None - CLI providers don't support authentication. - """ - - provider_type: Literal["cli"] = "cli" - command_name: str - env_vars: Optional[Dict[str, str]] = Field(default=None, description="Environment variables to set when executing the command") - working_dir: Optional[str] = Field(default=None, description="Working directory for command execution") - auth: None = None - -class WebSocketProvider(Provider): - """Provider configuration for WebSocket-based tools. (WIP) - - Enables bidirectional real-time communication with WebSocket servers. - Supports custom protocols, keep-alive functionality, and authentication. - - Attributes: - provider_type: Always "websocket" for WebSocket providers. - url: The WebSocket endpoint URL (ws:// or wss://). - protocol: Optional WebSocket sub-protocol to request. - keep_alive: Whether to maintain the connection with keep-alive messages. - auth: Optional authentication configuration. - headers: Optional static headers for the WebSocket handshake. - header_fields: List of tool argument names to map to headers during handshake. - """ - - provider_type: Literal["websocket"] = "websocket" - url: str - protocol: Optional[str] = None - keep_alive: bool = True - auth: Optional[Auth] = None - headers: Optional[Dict[str, str]] = None - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers for the initial connection.") - -class GRPCProvider(Provider): - """Provider configuration for gRPC (Google Remote Procedure Call) tools. (WIP) - - Enables communication with gRPC services using the Protocol Buffers - serialization format. Supports both secure (TLS) and insecure connections. - - Attributes: - provider_type: Always "grpc" for gRPC providers. - host: The hostname or IP address of the gRPC server. - port: The port number of the gRPC server. - service_name: The name of the gRPC service to call. - method_name: The name of the gRPC method to invoke. - use_ssl: Whether to use SSL/TLS for secure connections. - auth: Optional authentication configuration. - """ - - provider_type: Literal["grpc"] = "grpc" - host: str - port: int - service_name: str - method_name: str - use_ssl: bool = False - auth: Optional[Auth] = None - -class GraphQLProvider(Provider): - """Provider configuration for GraphQL-based tools. - - Enables communication with GraphQL endpoints supporting queries, mutations, - and subscriptions. Provides flexible query execution with custom headers - and authentication. - - Attributes: - provider_type: Always "graphql" for GraphQL providers. - url: The GraphQL endpoint URL. - operation_type: The type of GraphQL operation (query, mutation, subscription). - operation_name: Optional name for the GraphQL operation. - auth: Optional authentication configuration. - headers: Optional static headers to include in requests. - header_fields: List of tool argument names to map to HTTP request headers. - """ - - provider_type: Literal["graphql"] = "graphql" - url: str - operation_type: Literal["query", "mutation", "subscription"] = "query" - operation_name: Optional[str] = None - auth: Optional[Auth] = None - headers: Optional[Dict[str, str]] = None - header_fields: Optional[List[str]] = Field(default=None, description="List of input fields to be sent as request headers for the initial connection.") - -class TCPProvider(Provider): - """Provider configuration for raw TCP socket tools. - - Enables direct communication with TCP servers using custom protocols. - Supports flexible request formatting, response decoding, and multiple - framing strategies for message boundaries. - - Request Data Handling: - - 'json' format: Arguments formatted as JSON object - - 'text' format: Template-based with UTCP_ARG_argname_UTCP_ARG placeholders - - Response Data Handling: - - If response_byte_format is None: Returns raw bytes - - If response_byte_format is encoding string: Decodes bytes to text - - TCP Stream Framing Options: - 1. Length-prefix: Set framing_strategy='length_prefix' + length_prefix_bytes - 2. Delimiter-based: Set framing_strategy='delimiter' + message_delimiter - 3. Fixed-length: Set framing_strategy='fixed_length' + fixed_message_length - 4. Stream-based: Set framing_strategy='stream' (reads until connection closes) - - Attributes: - provider_type: Always "tcp" for TCP providers. - host: The hostname or IP address of the TCP server. - port: The port number of the TCP server. - request_data_format: Format for request data ('json' or 'text'). - request_data_template: Template string for 'text' format with placeholders. - response_byte_format: Encoding for response decoding (None for raw bytes). - framing_strategy: Method for detecting message boundaries. - length_prefix_bytes: Number of bytes for length prefix (1, 2, 4, or 8). - length_prefix_endian: Byte order for length prefix ('big' or 'little'). - message_delimiter: Delimiter string for message boundaries. - fixed_message_length: Fixed length in bytes for each message. - max_response_size: Maximum bytes to read for stream-based framing. - timeout: Connection timeout in milliseconds. - auth: Always None - TCP providers don't support authentication. - """ - - provider_type: Literal["tcp"] = "tcp" - host: str - port: int - request_data_format: Literal["json", "text"] = "json" - request_data_template: Optional[str] = None - response_byte_format: Optional[str] = Field(default="utf-8", description="Encoding to decode response bytes. If None, returns raw bytes.") - # TCP Framing Strategy - framing_strategy: Literal["length_prefix", "delimiter", "fixed_length", "stream"] = Field( - default="stream", - description="Strategy for framing TCP messages" - ) - # Length-prefix framing options - length_prefix_bytes: Literal[1, 2, 4, 8] = Field( - default=4, - description="Number of bytes for length prefix (1, 2, 4, or 8). Used with 'length_prefix' framing." - ) - length_prefix_endian: Literal["big", "little"] = Field( - default="big", - description="Byte order for length prefix. Used with 'length_prefix' framing." - ) - # Delimiter-based framing options - message_delimiter: str = Field( - default='\\x00', - description="Delimiter to detect end of TCP response (e.g., '\\n', '\\r\\n', '\\x00'). Used with 'delimiter' framing." - ) - # Fixed-length framing options - fixed_message_length: Optional[int] = Field( - default=None, - description="Fixed length of each message in bytes. Used with 'fixed_length' framing." - ) - # Stream-based options - max_response_size: int = Field( - default=65536, - description="Maximum bytes to read from TCP stream. Used with 'stream' framing." - ) - timeout: int = 30000 - auth: None = None - -class UDPProvider(Provider): - """Provider configuration for UDP (User Datagram Protocol) socket tools. - - Enables communication with UDP servers using the connectionless UDP protocol. - Supports flexible request formatting, response decoding, and multi-datagram - response handling. - - Request Data Handling: - - 'json' format: Arguments formatted as JSON object - - 'text' format: Template-based with UTCP_ARG_argname_UTCP_ARG placeholders - - Response Data Handling: - - If response_byte_format is None: Returns raw bytes - - If response_byte_format is encoding string: Decodes bytes to text - - Attributes: - provider_type: Always "udp" for UDP providers. - host: The hostname or IP address of the UDP server. - port: The port number of the UDP server. - number_of_response_datagrams: Expected number of response datagrams (0 for no response). - request_data_format: Format for request data ('json' or 'text'). - request_data_template: Template string for 'text' format with placeholders. - response_byte_format: Encoding for response decoding (None for raw bytes). - timeout: Request timeout in milliseconds. - auth: Always None - UDP providers don't support authentication. - """ - - provider_type: Literal["udp"] = "udp" - host: str - port: int - number_of_response_datagrams: int = 1 - request_data_format: Literal["json", "text"] = "json" - request_data_template: Optional[str] = None - response_byte_format: Optional[str] = Field(default="utf-8", description="Encoding to decode response bytes. If None, returns raw bytes.") - timeout: int = 30000 - auth: None = None - -class WebRTCProvider(Provider): - """Provider configuration for WebRTC (Web Real-Time Communication) tools. - - Enables peer-to-peer communication using WebRTC data channels. - Requires a signaling server to establish the initial connection. - - Attributes: - provider_type: Always "webrtc" for WebRTC providers. - signaling_server: URL of the signaling server for peer discovery. - peer_id: Unique identifier for this peer in the WebRTC network. - data_channel_name: Name of the data channel for tool communication. - auth: Always None - WebRTC providers don't support authentication. - """ - - provider_type: Literal["webrtc"] = "webrtc" - signaling_server: str - peer_id: str - data_channel_name: str = "tools" - auth: None = None - -class McpStdioServer(BaseModel): - """Configuration for an MCP server connected via stdio transport. - - Enables communication with Model Context Protocol servers through - standard input/output streams, typically used for local processes. - - Attributes: - transport: Always "stdio" for stdio-based MCP servers. - command: The command to execute to start the MCP server. - args: Optional command-line arguments for the MCP server. - env: Optional environment variables for the MCP server process. - """ - transport: Literal["stdio"] = "stdio" - command: str - args: Optional[List[str]] = [] - env: Optional[Dict[str, str]] = {} - -class McpHttpServer(BaseModel): - """Configuration for an MCP server connected via HTTP transport. - - Enables communication with Model Context Protocol servers through - HTTP connections, typically used for remote MCP services. - - Attributes: - transport: Always "http" for HTTP-based MCP servers. - url: The HTTP endpoint URL for the MCP server. - """ - transport: Literal["http"] = "http" - url: str - -McpServer: TypeAlias = Union[McpStdioServer, McpHttpServer] -"""Type alias for MCP server configurations. - -Union type for all supported MCP server transport configurations, -including both stdio and HTTP-based servers. -""" - -class McpConfig(BaseModel): - """Configuration container for multiple MCP servers. - - Holds a collection of named MCP server configurations, allowing - a single MCP provider to manage multiple server connections. - - Attributes: - mcpServers: Dictionary mapping server names to their configurations. - """ - - mcpServers: Dict[str, McpServer] - -class MCPProvider(Provider): - """Provider configuration for Model Context Protocol (MCP) tools. - - Enables communication with MCP servers that provide structured tool - interfaces. Supports both stdio (local process) and HTTP (remote) - transport methods. - - Attributes: - provider_type: Always "mcp" for MCP providers. - config: Configuration object containing MCP server definitions. - This follows the same format as the official MCP server configuration. - auth: Optional OAuth2 authentication for HTTP-based MCP servers. - """ - - provider_type: Literal["mcp"] = "mcp" - config: McpConfig - auth: Optional[OAuth2Auth] = None - - -class TextProvider(Provider): - """Provider configuration for text file-based tools. - - Reads tool definitions from local text files, useful for static tool - configurations or when tools generate output files at known locations. - - Use Cases: - - Static tool definitions from configuration files - - Tools that write results to predictable file locations - - Download manuals from a remote server to allow inspection of tools - before calling them and guarantee security for high-risk environments - - Attributes: - provider_type: Always "text" for text file providers. - file_path: Path to the file containing tool definitions. - auth: Always None - text providers don't support authentication. - """ - - provider_type: Literal["text"] = "text" - file_path: str = Field(..., description="The path to the file containing the tool definitions.") - auth: None = None - -ProviderUnion = Annotated[ - Union[ - HttpProvider, - SSEProvider, - StreamableHttpProvider, - CliProvider, - WebSocketProvider, - GRPCProvider, - GraphQLProvider, - TCPProvider, - UDPProvider, - WebRTCProvider, - MCPProvider, - TextProvider - ], - Field(discriminator="provider_type") -] -"""Discriminated union type for all UTCP provider configurations. - -This annotated union type includes all supported provider implementations, -using 'provider_type' as the discriminator field for automatic type -resolution during deserialization. - -Supported Provider Types: - - HttpProvider: RESTful HTTP/HTTPS APIs - - SSEProvider: Server-Sent Events streaming - - StreamableHttpProvider: HTTP Chunked Transfer Encoding - - CliProvider: Command Line Interface tools - - WebSocketProvider: Bidirectional WebSocket connections - - GRPCProvider: Google Remote Procedure Call - - GraphQLProvider: GraphQL query language - - TCPProvider: Raw TCP socket connections - - UDPProvider: User Datagram Protocol - - WebRTCProvider: Web Real-Time Communication - - MCPProvider: Model Context Protocol - - TextProvider: Text file-based providers -""" diff --git a/src/utcp/shared/utcp_manual.py b/src/utcp/shared/utcp_manual.py deleted file mode 100644 index 9a9b7e9..0000000 --- a/src/utcp/shared/utcp_manual.py +++ /dev/null @@ -1,74 +0,0 @@ -"""UTCP manual data structure for tool discovery. - -This module defines the UtcpManual model that standardizes the format for -tool provider responses during tool discovery. It serves as the contract -between tool providers and clients for sharing available tools and their -configurations. -""" - -from typing import List -from pydantic import BaseModel, ConfigDict -from utcp.shared.tool import Tool, ToolContext -from utcp.version import __version__ -class UtcpManual(BaseModel): - """Standard format for tool provider responses during discovery. - - Represents the complete set of tools available from a provider, along - with version information for compatibility checking. This format is - returned by tool providers when clients query for available tools - (e.g., through the `/utcp` endpoint or similar discovery mechanisms). - - The manual serves as the authoritative source of truth for what tools - a provider offers and how they should be invoked. - - Attributes: - version: UTCP protocol version supported by the provider. - Defaults to the current library version. - tools: List of available tools with their complete configurations - including input/output schemas, descriptions, and metadata. - - Example: - ```python - # Create a manual from registered tools - manual = UtcpManual.create() - - # Manual with specific tools - manual = UtcpManual( - version="1.0.0", - tools=[tool1, tool2, tool3] - ) - ``` - """ - version: str = __version__ - tools: List[Tool] - - model_config = ConfigDict(arbitrary_types_allowed=True) - - @staticmethod - def create(version: str = __version__) -> "UtcpManual": - """Create a UTCP manual from the global tool registry. - - Convenience method that creates a manual containing all tools - currently registered in the global ToolContext. This is typically - used by tool providers to generate their discovery response. - - Args: - version: UTCP protocol version to include in the manual. - Defaults to the current library version. - - Returns: - UtcpManual containing all registered tools and the specified version. - - Example: - ```python - # Create manual with default version - manual = UtcpManual.create() - - # Create manual with specific version - manual = UtcpManual.create(version="1.2.0") - ``` - """ - return UtcpManual( - version=version, - tools=ToolContext.get_tools() - ) diff --git a/tests/client/test_utcp_client.py b/tests/client/test_utcp_client.py deleted file mode 100644 index c1cd9f5..0000000 --- a/tests/client/test_utcp_client.py +++ /dev/null @@ -1,788 +0,0 @@ -import pytest -import pytest_asyncio -import asyncio -import json -import os -import tempfile -from typing import Dict, Any, List, Optional -from unittest.mock import MagicMock, AsyncMock, patch - -from utcp.client.utcp_client import UtcpClient, UtcpClientInterface -from utcp.client.utcp_client_config import UtcpClientConfig, UtcpVariableNotFound -from utcp.client.tool_repository import ToolRepository -from utcp.client.tool_repositories.in_mem_tool_repository import InMemToolRepository -from utcp.client.tool_search_strategy import ToolSearchStrategy -from utcp.client.tool_search_strategies.tag_search import TagSearchStrategy -from utcp.client.variable_substitutor import VariableSubstitutor, DefaultVariableSubstitutor -from utcp.shared.tool import Tool, ToolInputOutputSchema -from utcp.shared.provider import ( - Provider, HttpProvider, CliProvider, MCPProvider, TextProvider, - McpConfig, McpStdioServer, McpHttpServer -) -from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth - - -class MockToolRepository(ToolRepository): - """Mock tool repository for testing.""" - - def __init__(self): - self.providers: Dict[str, Provider] = {} - self.tools: Dict[str, Tool] = {} - self.provider_tools: Dict[str, List[Tool]] = {} - - async def save_provider_with_tools(self, provider: Provider, tools: List[Tool]) -> None: - self.providers[provider.name] = provider - self.provider_tools[provider.name] = tools - for tool in tools: - self.tools[tool.name] = tool - - async def remove_provider(self, provider_name: str) -> None: - if provider_name not in self.providers: - raise ValueError(f"Provider not found: {provider_name}") - # Remove tools associated with provider - if provider_name in self.provider_tools: - for tool in self.provider_tools[provider_name]: - if tool.name in self.tools: - del self.tools[tool.name] - del self.provider_tools[provider_name] - del self.providers[provider_name] - - async def remove_tool(self, tool_name: str) -> None: - if tool_name not in self.tools: - raise ValueError(f"Tool not found: {tool_name}") - del self.tools[tool_name] - # Remove from provider_tools - for provider_name, tools in self.provider_tools.items(): - self.provider_tools[provider_name] = [t for t in tools if t.name != tool_name] - - async def get_tool(self, tool_name: str) -> Optional[Tool]: - return self.tools.get(tool_name) - - async def get_tools(self) -> List[Tool]: - return list(self.tools.values()) - - async def get_tools_by_provider(self, provider_name: str) -> Optional[List[Tool]]: - return self.provider_tools.get(provider_name) - - async def get_provider(self, provider_name: str) -> Optional[Provider]: - return self.providers.get(provider_name) - - async def get_providers(self) -> List[Provider]: - return list(self.providers.values()) - - -class MockToolSearchStrategy(ToolSearchStrategy): - """Mock search strategy for testing.""" - - def __init__(self, tool_repository: ToolRepository): - self.tool_repository = tool_repository - - async def search_tools(self, query: str, limit: int = 10) -> List[Tool]: - tools = await self.tool_repository.get_tools() - # Simple mock search: return tools that contain the query in name or description - matched_tools = [ - tool for tool in tools - if query.lower() in tool.name.lower() or query.lower() in tool.description.lower() - ] - return matched_tools[:limit] if limit > 0 else matched_tools - - -class MockTransport: - """Mock transport for testing.""" - - def __init__(self, tools: List[Tool] = None, call_result: Any = "mock_result"): - self.tools = tools or [] - self.call_result = call_result - self.registered_providers = [] - self.deregistered_providers = [] - self.tool_calls = [] - - async def register_tool_provider(self, provider: Provider) -> List[Tool]: - self.registered_providers.append(provider) - return self.tools - - async def deregister_tool_provider(self, provider: Provider) -> None: - self.deregistered_providers.append(provider) - - async def call_tool(self, tool_name: str, arguments: Dict[str, Any], tool_provider: Provider) -> Any: - self.tool_calls.append((tool_name, arguments, tool_provider)) - return self.call_result - - -@pytest_asyncio.fixture -async def mock_tool_repository(): - """Create a mock tool repository.""" - return MockToolRepository() - - -@pytest_asyncio.fixture -async def mock_search_strategy(mock_tool_repository): - """Create a mock search strategy.""" - return MockToolSearchStrategy(mock_tool_repository) - - -@pytest_asyncio.fixture -async def sample_tools(): - """Create sample tools for testing.""" - http_provider = HttpProvider( - name="test_http_provider", - url="https://api.example.com/tool", - http_method="POST" - ) - - cli_provider = CliProvider( - name="test_cli_provider", - command_name="echo" - ) - - return [ - Tool( - name="http_tool", - description="HTTP test tool", - inputs=ToolInputOutputSchema( - type="object", - properties={"param1": {"type": "string", "description": "Test parameter"}}, - required=["param1"] - ), - outputs=ToolInputOutputSchema( - type="object", - properties={"result": {"type": "string", "description": "Test result"}} - ), - tags=["http", "test"], - tool_provider=http_provider - ), - Tool( - name="cli_tool", - description="CLI test tool", - inputs=ToolInputOutputSchema( - type="object", - properties={"command": {"type": "string", "description": "Command to execute"}}, - required=["command"] - ), - outputs=ToolInputOutputSchema( - type="object", - properties={"output": {"type": "string", "description": "Command output"}} - ), - tags=["cli", "test"], - tool_provider=cli_provider - ) - ] - - -@pytest_asyncio.fixture -async def utcp_client(mock_tool_repository, mock_search_strategy): - """Create a UtcpClient instance with mocked dependencies.""" - config = UtcpClientConfig() - variable_substitutor = DefaultVariableSubstitutor() - - client = UtcpClient(config, mock_tool_repository, mock_search_strategy, variable_substitutor) - - # Clear the repository before each test to ensure clean state - client.tool_repository.providers.clear() - client.tool_repository.tools.clear() - client.tool_repository.provider_tools.clear() - - return client - - -class TestUtcpClientInterface: - """Test the UtcpClientInterface abstract methods.""" - - def test_interface_is_abstract(self): - """Test that UtcpClientInterface cannot be instantiated directly.""" - with pytest.raises(TypeError): - UtcpClientInterface() - - def test_utcp_client_implements_interface(self): - """Test that UtcpClient properly implements the interface.""" - assert issubclass(UtcpClient, UtcpClientInterface) - - -class TestUtcpClient: - """Test the UtcpClient implementation.""" - - @pytest.mark.asyncio - async def test_init(self, mock_tool_repository, mock_search_strategy): - """Test UtcpClient initialization.""" - config = UtcpClientConfig() - variable_substitutor = DefaultVariableSubstitutor() - - client = UtcpClient(config, mock_tool_repository, mock_search_strategy, variable_substitutor) - - assert client.config is config - assert client.tool_repository is mock_tool_repository - assert client.search_strategy is mock_search_strategy - assert client.variable_substitutor is variable_substitutor - - @pytest.mark.asyncio - async def test_create_with_defaults(self): - """Test creating UtcpClient with default parameters.""" - with patch.object(UtcpClient, 'load_providers', new_callable=AsyncMock): - client = await UtcpClient.create() - - assert isinstance(client.config, UtcpClientConfig) - assert isinstance(client.tool_repository, InMemToolRepository) - assert isinstance(client.search_strategy, TagSearchStrategy) - assert isinstance(client.variable_substitutor, DefaultVariableSubstitutor) - - @pytest.mark.asyncio - async def test_create_with_dict_config(self): - """Test creating UtcpClient with dictionary config.""" - config_dict = { - "variables": {"TEST_VAR": "test_value"}, - "providers_file_path": "test_providers.json" - } - - with patch.object(UtcpClient, 'load_providers', new_callable=AsyncMock): - client = await UtcpClient.create(config=config_dict) - - assert client.config.variables == {"TEST_VAR": "test_value"} - assert client.config.providers_file_path == "test_providers.json" - - @pytest.mark.asyncio - async def test_create_with_utcp_config(self): - """Test creating UtcpClient with UtcpClientConfig object.""" - config = UtcpClientConfig( - variables={"TEST_VAR": "test_value"}, - providers_file_path="test_providers.json" - ) - - with patch.object(UtcpClient, 'load_providers', new_callable=AsyncMock): - client = await UtcpClient.create(config=config) - - assert client.config is config - - @pytest.mark.asyncio - async def test_register_tool_provider(self, utcp_client, sample_tools): - """Test registering a tool provider.""" - http_provider = HttpProvider( - name="test_provider", - url="https://api.example.com/tool", - http_method="POST" - ) - - # Mock the transport - mock_transport = MockTransport(sample_tools[:1]) # Return first tool - utcp_client.transports["http"] = mock_transport - - tools = await utcp_client.register_tool_provider(http_provider) - - assert len(tools) == 1 - assert tools[0].name == "test_provider.http_tool" # Should be prefixed - # Check that the registered provider has the expected properties - registered_provider = mock_transport.registered_providers[0] - assert registered_provider.name == "test_provider" - assert registered_provider.url == "https://api.example.com/tool" - assert registered_provider.http_method == "POST" - - # Verify tool was saved in repository - saved_tool = await utcp_client.tool_repository.get_tool("test_provider.http_tool") - assert saved_tool is not None - - @pytest.mark.asyncio - async def test_register_tool_provider_unsupported_type(self, utcp_client): - """Test registering a tool provider with unsupported type.""" - # Create a provider with a supported type but then modify it - provider = HttpProvider( - name="test_provider", - url="https://example.com", - http_method="GET" - ) - - # Simulate an unsupported type by removing it from transports - original_transports = utcp_client.transports.copy() - del utcp_client.transports["http"] - - try: - with pytest.raises(ValueError, match="Provider type not supported: http"): - await utcp_client.register_tool_provider(provider) - finally: - # Restore original transports - utcp_client.transports = original_transports - - @pytest.mark.asyncio - async def test_register_tool_provider_name_sanitization(self, utcp_client, sample_tools): - """Test that provider names are sanitized.""" - provider = HttpProvider( - name="test-provider.with/special@chars", - url="https://api.example.com/tool", - http_method="POST" - ) - - mock_transport = MockTransport(sample_tools[:1]) - utcp_client.transports["http"] = mock_transport - - tools = await utcp_client.register_tool_provider(provider) - - # Name should be sanitized - assert provider.name == "test_provider_with_special_chars" - assert tools[0].name == "test_provider_with_special_chars.http_tool" - - @pytest.mark.asyncio - async def test_deregister_tool_provider(self, utcp_client, sample_tools): - """Test deregistering a tool provider.""" - provider = HttpProvider( - name="test_provider", - url="https://api.example.com/tool", - http_method="POST" - ) - - mock_transport = MockTransport(sample_tools[:1]) - utcp_client.transports["http"] = mock_transport - - # First register the provider - await utcp_client.register_tool_provider(provider) - - # Then deregister it - await utcp_client.deregister_tool_provider("test_provider") - - # Verify provider was removed from repository - saved_provider = await utcp_client.tool_repository.get_provider("test_provider") - assert saved_provider is None - - # Verify transport deregister was called - assert len(mock_transport.deregistered_providers) == 1 - - @pytest.mark.asyncio - async def test_deregister_nonexistent_provider(self, utcp_client): - """Test deregistering a non-existent provider.""" - with pytest.raises(ValueError, match="Provider not found: nonexistent"): - await utcp_client.deregister_tool_provider("nonexistent") - - @pytest.mark.asyncio - async def test_call_tool(self, utcp_client, sample_tools): - """Test calling a tool.""" - provider = HttpProvider( - name="test_provider", - url="https://api.example.com/tool", - http_method="POST" - ) - - mock_transport = MockTransport(sample_tools[:1], "test_result") - utcp_client.transports["http"] = mock_transport - - # Register the provider first - await utcp_client.register_tool_provider(provider) - - # Call the tool - result = await utcp_client.call_tool("test_provider.http_tool", {"param1": "value1"}) - - assert result == "test_result" - assert len(mock_transport.tool_calls) == 1 - assert mock_transport.tool_calls[0][0] == "test_provider.http_tool" - assert mock_transport.tool_calls[0][1] == {"param1": "value1"} - - @pytest.mark.asyncio - async def test_call_tool_nonexistent_provider(self, utcp_client): - """Test calling a tool with nonexistent provider.""" - with pytest.raises(ValueError, match="Provider not found: nonexistent"): - await utcp_client.call_tool("nonexistent.tool", {"param": "value"}) - - @pytest.mark.asyncio - async def test_call_tool_nonexistent_tool(self, utcp_client, sample_tools): - """Test calling a nonexistent tool.""" - provider = HttpProvider( - name="test_provider", - url="https://api.example.com/tool", - http_method="POST" - ) - - mock_transport = MockTransport(sample_tools[:1]) - utcp_client.transports["http"] = mock_transport - - # Register the provider first - await utcp_client.register_tool_provider(provider) - - with pytest.raises(ValueError, match="Tool not found: test_provider.nonexistent"): - await utcp_client.call_tool("test_provider.nonexistent", {"param": "value"}) - - @pytest.mark.asyncio - async def test_search_tools(self, utcp_client, sample_tools): - """Test searching for tools.""" - # Add tools to the search strategy's repository - for i, tool in enumerate(sample_tools): - tool.name = f"provider_{i}.{tool.name}" - await utcp_client.tool_repository.save_provider_with_tools( - tool.tool_provider, [tool] - ) - - # Search for tools - results = await utcp_client.search_tools("http", limit=10) - - # Should find the HTTP tool - assert len(results) == 1 - assert "http" in results[0].name.lower() or "http" in results[0].description.lower() - - @pytest.mark.asyncio - async def test_get_required_variables_for_manual_and_tools(self, utcp_client): - """Test getting required variables for a provider.""" - provider = HttpProvider( - name="test_provider", - url="https://api.example.com/$API_URL", - http_method="POST", - auth=ApiKeyAuth(api_key="$API_KEY", var_name="Authorization") - ) - - # Mock the variable substitutor - mock_substitutor = MagicMock() - mock_substitutor.find_required_variables.return_value = ["API_URL", "API_KEY"] - mock_substitutor.substitute.return_value = provider.model_dump() # Return the original dict - utcp_client.variable_substitutor = mock_substitutor - - variables = await utcp_client.get_required_variables_for_manual_and_tools(provider) - - assert variables == ["API_URL", "API_KEY"] - mock_substitutor.find_required_variables.assert_called_once() - - @pytest.mark.asyncio - async def test_get_required_variables_for_tool(self, utcp_client, sample_tools): - """Test getting required variables for a tool.""" - provider = HttpProvider( - name="test_provider", - url="https://api.example.com/$API_URL", - http_method="POST" - ) - - tool = sample_tools[0] - tool.name = "test_provider.http_tool" - tool.tool_provider = provider - - # Add tool to repository - await utcp_client.tool_repository.save_provider_with_tools(provider, [tool]) - - # Mock the variable substitutor - mock_substitutor = MagicMock() - mock_substitutor.find_required_variables.return_value = ["API_URL"] - utcp_client.variable_substitutor = mock_substitutor - - variables = await utcp_client.get_required_variables_for_tool("test_provider.http_tool") - - assert variables == ["API_URL"] - mock_substitutor.find_required_variables.assert_called_once() - - @pytest.mark.asyncio - async def test_get_required_variables_for_nonexistent_tool(self, utcp_client): - """Test getting required variables for a nonexistent tool.""" - with pytest.raises(ValueError, match="Tool not found: nonexistent.tool"): - await utcp_client.get_required_variables_for_tool("nonexistent.tool") - - -class TestUtcpClientProviderLoading: - """Test provider loading functionality.""" - - @pytest.mark.asyncio - async def test_load_providers_from_file(self, utcp_client): - """Test loading providers from a JSON file.""" - # Create a temporary providers file with array format (as expected by load_providers) - providers_data = [ - { - "name": "http_provider", - "provider_type": "http", - "url": "https://api.example.com/tools", - "http_method": "GET" - }, - { - "name": "cli_provider", - "provider_type": "cli", - "command_name": "echo" - } - ] - - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump(providers_data, f) - temp_file = f.name - - try: - # Mock the transports - mock_http_transport = MockTransport([]) - mock_cli_transport = MockTransport([]) - utcp_client.transports["http"] = mock_http_transport - utcp_client.transports["cli"] = mock_cli_transport - - # Load providers - providers = await utcp_client.load_providers(temp_file) - - assert len(providers) == 2 - assert len(mock_http_transport.registered_providers) == 1 - assert len(mock_cli_transport.registered_providers) == 1 - - finally: - os.unlink(temp_file) - - @pytest.mark.asyncio - async def test_load_providers_file_not_found(self, utcp_client): - """Test loading providers from a non-existent file.""" - with pytest.raises(FileNotFoundError): - await utcp_client.load_providers("nonexistent.json") - - @pytest.mark.asyncio - async def test_load_providers_invalid_json(self, utcp_client): - """Test loading providers from invalid JSON file.""" - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - f.write("invalid json content") - temp_file = f.name - - try: - with pytest.raises(ValueError, match="Invalid JSON in providers file"): - await utcp_client.load_providers(temp_file) - finally: - os.unlink(temp_file) - - @pytest.mark.asyncio - async def test_load_providers_with_variables(self, utcp_client): - """Test loading providers with variable substitution.""" - providers_data = [ - { - "name": "http_provider", - "provider_type": "http", - "url": "$BASE_URL/tools", - "http_method": "GET", - "auth": { - "auth_type": "api_key", - "api_key": "$API_KEY", - "var_name": "Authorization" - } - } - ] - - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump(providers_data, f) - temp_file = f.name - - try: - # Setup client with variables (need provider prefixed variables) - utcp_client.config.variables = { - "http__provider_BASE_URL": "https://api.example.com", - "http__provider_API_KEY": "secret_key" - } - - # Mock the transport - mock_transport = MockTransport([]) - utcp_client.transports["http"] = mock_transport - - # Load providers - providers = await utcp_client.load_providers(temp_file) - - assert len(providers) == 1 - # Check that the registered provider has substituted values - registered_provider = mock_transport.registered_providers[0] - assert registered_provider.url == "https://api.example.com/tools" - assert registered_provider.auth.api_key == "secret_key" - - finally: - os.unlink(temp_file) - - @pytest.mark.asyncio - async def test_load_providers_missing_variable(self, utcp_client): - """Test loading providers with missing variable.""" - providers_data = [ - { - "name": "http_provider", - "provider_type": "http", - "url": "$MISSING_VAR/tools", - "http_method": "GET" - } - ] - - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump(providers_data, f) - temp_file = f.name - - try: - # Mock transport to avoid registration issues - utcp_client.transports["http"] = MockTransport([]) - - # The load_providers method catches exceptions and returns empty list - # So we need to check the registration directly which will raise the exception - provider_data = { - "name": "http_provider", - "provider_type": "http", - "url": "$MISSING_VAR/tools", - "http_method": "GET" - } - provider = HttpProvider.model_validate(provider_data) - - with pytest.raises(UtcpVariableNotFound, match="Variable http__provider_MISSING_VAR"): - await utcp_client.register_tool_provider(provider) - finally: - os.unlink(temp_file) - - -class TestUtcpClientTransports: - """Test transport-related functionality.""" - - def test_default_transports_initialized(self, utcp_client): - """Test that default transports are properly initialized.""" - expected_transport_types = [ - "http", "cli", "sse", "http_stream", "mcp", "text", "graphql", "tcp", "udp" - ] - - for transport_type in expected_transport_types: - assert transport_type in utcp_client.transports - assert utcp_client.transports[transport_type] is not None - - @pytest.mark.asyncio - async def test_variable_substitution(self, utcp_client): - """Test variable substitution in providers.""" - provider = HttpProvider( - name="test_provider", - url="$BASE_URL/api", - http_method="POST", - auth=ApiKeyAuth(api_key="$API_KEY", var_name="Authorization") - ) - - # Set up variables with provider prefix - utcp_client.config.variables = { - "test__provider_BASE_URL": "https://api.example.com", - "test__provider_API_KEY": "secret_key" - } - - substituted_provider = utcp_client._substitute_provider_variables(provider, "test_provider") - - assert substituted_provider.url == "https://api.example.com/api" - assert substituted_provider.auth.api_key == "secret_key" - - @pytest.mark.asyncio - async def test_variable_substitution_missing_variable(self, utcp_client): - """Test variable substitution with missing variable.""" - provider = HttpProvider( - name="test_provider", - url="$MISSING_VAR/api", - http_method="POST" - ) - - with pytest.raises(UtcpVariableNotFound, match="Variable test__provider_MISSING_VAR"): - utcp_client._substitute_provider_variables(provider, "test_provider") - - -class TestUtcpClientEdgeCases: - """Test edge cases and error conditions.""" - - @pytest.mark.asyncio - async def test_empty_provider_file(self, utcp_client): - """Test loading an empty provider file.""" - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump([], f) # Empty array instead of empty object - temp_file = f.name - - try: - providers = await utcp_client.load_providers(temp_file) - assert providers == [] - finally: - os.unlink(temp_file) - - @pytest.mark.asyncio - async def test_register_provider_with_existing_name(self, utcp_client, sample_tools): - """Test registering a provider with an existing name should raise an error.""" - provider1 = HttpProvider( - name="duplicate_name", - url="https://api.example1.com/tool", - http_method="POST" - ) - provider2 = HttpProvider( - name="duplicate_name", - url="https://api.example2.com/tool", - http_method="GET" - ) - - mock_transport = MockTransport(sample_tools[:1]) - utcp_client.transports["http"] = mock_transport - - # Register first provider - await utcp_client.register_tool_provider(provider1) - - # Attempting to register second provider with same name should raise an error - with pytest.raises(ValueError, match="Provider duplicate_name already registered"): - await utcp_client.register_tool_provider(provider2) - - # Should still have the first provider - saved_provider = await utcp_client.tool_repository.get_provider("duplicate_name") - assert saved_provider.url == "https://api.example1.com/tool" - assert saved_provider.http_method == "POST" - - @pytest.mark.asyncio - async def test_complex_mcp_provider(self, utcp_client): - """Test loading a complex MCP provider configuration.""" - providers_data = [ - { - "name": "mcp_provider", - "provider_type": "mcp", - "config": { - "mcpServers": { - "stdio_server": { - "transport": "stdio", - "command": "python", - "args": ["-m", "test_server"], - "env": {"TEST_VAR": "test_value"} - }, - "http_server": { - "transport": "http", - "url": "http://localhost:8000/mcp" - } - } - } - } - ] - - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump(providers_data, f) - temp_file = f.name - - try: - # Mock the MCP transport - mock_transport = MockTransport([]) - utcp_client.transports["mcp"] = mock_transport - - providers = await utcp_client.load_providers(temp_file) - - assert len(providers) == 1 - provider = providers[0] - assert isinstance(provider, MCPProvider) - assert len(provider.config.mcpServers) == 2 - assert "stdio_server" in provider.config.mcpServers - assert "http_server" in provider.config.mcpServers - - finally: - os.unlink(temp_file) - - @pytest.mark.asyncio - async def test_text_transport_configuration(self, utcp_client): - """Test TextTransport base path configuration.""" - # Create a temporary directory structure - with tempfile.TemporaryDirectory() as temp_dir: - providers_file = os.path.join(temp_dir, "providers.json") - - with open(providers_file, 'w') as f: - json.dump([], f) # Empty array - - # Create client with providers file path - config = UtcpClientConfig(providers_file_path=providers_file) - - with patch.object(UtcpClient, 'load_providers', new_callable=AsyncMock): - client = await UtcpClient.create(config=config) - - # Check that TextTransport was configured with the correct base path - text_transport = client.transports["text"] - assert hasattr(text_transport, 'base_path') - assert text_transport.base_path == temp_dir - - @pytest.mark.asyncio - async def test_load_providers_wrong_format(self, utcp_client): - """Test loading providers with wrong JSON format (object instead of array).""" - providers_data = { - "http_provider": { - "provider_type": "http", - "url": "https://api.example.com/tools", - "http_method": "GET" - } - } - - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump(providers_data, f) - temp_file = f.name - - try: - with pytest.raises(ValueError, match="Providers file must contain a JSON array at the root level"): - await utcp_client.load_providers(temp_file) - finally: - os.unlink(temp_file) diff --git a/tests/client/transport_interfaces/test_graphql_transport.py b/tests/client/transport_interfaces/test_graphql_transport.py deleted file mode 100644 index 9bd020f..0000000 --- a/tests/client/transport_interfaces/test_graphql_transport.py +++ /dev/null @@ -1,129 +0,0 @@ -import pytest -import pytest_asyncio -import json -from aiohttp import web -from utcp.client.transport_interfaces.graphql_transport import GraphQLClientTransport -from utcp.shared.provider import GraphQLProvider -from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth - - -@pytest_asyncio.fixture -async def graphql_app(): - async def graphql_handler(request): - body = await request.json() - query = body.get("query", "") - variables = body.get("variables", {}) - # Introspection query (minimal response) - if "__schema" in query: - return web.json_response({ - "data": { - "__schema": { - "queryType": {"name": "Query"}, - "mutationType": {"name": "Mutation"}, - "subscriptionType": None, - "types": [ - {"kind": "OBJECT", "name": "Query", "fields": [ - {"name": "hello", "args": [{"name": "name", "type": {"kind": "SCALAR", "name": "String"}, "defaultValue": None}], "type": {"kind": "SCALAR", "name": "String"}, "isDeprecated": False, "deprecationReason": None} - ], "interfaces": []}, - {"kind": "OBJECT", "name": "Mutation", "fields": [ - {"name": "add", "args": [ - {"name": "a", "type": {"kind": "SCALAR", "name": "Int"}, "defaultValue": None}, - {"name": "b", "type": {"kind": "SCALAR", "name": "Int"}, "defaultValue": None} - ], "type": {"kind": "SCALAR", "name": "Int"}, "isDeprecated": False, "deprecationReason": None} - ], "interfaces": []}, - {"kind": "SCALAR", "name": "String"}, - {"kind": "SCALAR", "name": "Int"}, - {"kind": "SCALAR", "name": "Boolean"} - ], - "directives": [] - } - } - }) - # hello query - if "hello" in query: - name = variables.get("name", "world") - return web.json_response({"data": {"hello": f"Hello, {name}!"}}) - # add mutation - if "add" in query: - a = variables.get("a", 0) - b = variables.get("b", 0) - return web.json_response({"data": {"add": a + b}}) - # fallback - return web.json_response({"data": {}}, status=200) - - app = web.Application() - app.router.add_post("/graphql", graphql_handler) - return app - -@pytest_asyncio.fixture -async def aiohttp_graphql_client(aiohttp_client, graphql_app): - return await aiohttp_client(graphql_app) - -@pytest_asyncio.fixture -def transport(): - return GraphQLClientTransport() - -@pytest_asyncio.fixture -def provider(aiohttp_graphql_client): - return GraphQLProvider( - name="test-graphql-provider", - url=str(aiohttp_graphql_client.make_url("/graphql")), - headers={}, - ) - -@pytest.mark.asyncio -async def test_register_tool_provider_discovers_tools(transport, provider): - tools = await transport.register_tool_provider(provider) - tool_names = [tool.name for tool in tools] - assert "hello" in tool_names - assert "add" in tool_names - -@pytest.mark.asyncio -async def test_call_tool_query(transport, provider): - result = await transport.call_tool("hello", {"name": "Alice"}, provider) - assert result["hello"] == "Hello, Alice!" - -@pytest.mark.asyncio -async def test_call_tool_mutation(transport, provider): - provider.operation_type = "mutation" - mutation = ''' - mutation ($a: Int, $b: Int) { - add(a: $a, b: $b) - } - ''' - result = await transport.call_tool("add", {"a": 2, "b": 3}, provider, query=mutation) - assert result["add"] == 5 - -@pytest.mark.asyncio -async def test_call_tool_api_key(transport, provider): - provider.headers = {} - provider.auth = ApiKeyAuth(var_name="X-API-Key", api_key="test-key") - result = await transport.call_tool("hello", {"name": "Bob"}, provider) - assert result["hello"] == "Hello, Bob!" - -@pytest.mark.asyncio -async def test_call_tool_basic_auth(transport, provider): - provider.headers = {} - provider.auth = BasicAuth(username="user", password="pass") - result = await transport.call_tool("hello", {"name": "Eve"}, provider) - assert result["hello"] == "Hello, Eve!" - -@pytest.mark.asyncio -async def test_call_tool_oauth2(monkeypatch, transport, provider): - async def fake_oauth2(auth): - return "fake-token" - transport._handle_oauth2 = fake_oauth2 - provider.headers = {} - provider.auth = OAuth2Auth(token_url="http://fake/token", client_id="id", client_secret="secret", scope="scope") - result = await transport.call_tool("hello", {"name": "Zoe"}, provider) - assert result["hello"] == "Hello, Zoe!" - -@pytest.mark.asyncio -async def test_enforce_https_or_localhost_raises(transport, provider): - provider.url = "http://evil.com/graphql" - with pytest.raises(ValueError): - await transport.call_tool("hello", {"name": "Mallory"}, provider) - -@pytest.mark.asyncio -async def test_deregister_tool_provider_noop(transport, provider): - await transport.deregister_tool_provider(provider) \ No newline at end of file diff --git a/tests/client/transport_interfaces/test_mcp_transport.py b/tests/client/transport_interfaces/test_mcp_transport.py deleted file mode 100644 index b84a67a..0000000 --- a/tests/client/transport_interfaces/test_mcp_transport.py +++ /dev/null @@ -1,132 +0,0 @@ -import sys -import pytest -import pytest_asyncio -import asyncio - -from utcp.client.transport_interfaces.mcp_transport import MCPTransport -from utcp.shared.provider import MCPProvider, McpConfig, McpStdioServer - -SERVER_NAME = "mock_stdio_server" - -@pytest_asyncio.fixture -def mcp_provider() -> MCPProvider: - """Provides an MCPProvider configured to run the mock stdio server.""" - server_config = McpStdioServer( - command=sys.executable, - args=["tests/client/transport_interfaces/mock_mcp_server.py"], - ) - return MCPProvider( - name="mock_mcp_provider", - provider_type="mcp", - config=McpConfig(mcpServers={SERVER_NAME: server_config}) - ) - -@pytest_asyncio.fixture -async def transport() -> MCPTransport: - """Provides a clean MCPTransport instance.""" - t = MCPTransport() - yield t - await t.close() - -@pytest.mark.asyncio -async def test_register_provider_discovers_tools(transport: MCPTransport, mcp_provider: MCPProvider): - """Verify that registering a provider discovers the correct tools.""" - tools = await transport.register_tool_provider(mcp_provider) - assert len(tools) == 4 - - # Find the echo tool - echo_tool = next((tool for tool in tools if tool.name == "echo"), None) - assert echo_tool is not None - assert "echoes back its input" in echo_tool.description - - # Check for other tools - tool_names = [tool.name for tool in tools] - assert "greet" in tool_names - assert "list_items" in tool_names - assert "add_numbers" in tool_names - -@pytest.mark.asyncio -async def test_call_tool_succeeds(transport: MCPTransport, mcp_provider: MCPProvider): - """Verify a successful tool call after registration.""" - await transport.register_tool_provider(mcp_provider) - - result = await transport.call_tool("echo", {"message": "test"}, mcp_provider) - - assert result == {"reply": "you said: test"} - -@pytest.mark.asyncio -async def test_call_tool_works_without_register(transport: MCPTransport, mcp_provider: MCPProvider): - """Verify that calling a tool works without prior registration in session-per-operation mode.""" - # In session-per-operation mode, registration is not required - result = await transport.call_tool("echo", {"message": "test"}, mcp_provider) - assert result == {"reply": "you said: test"} - -@pytest.mark.asyncio -async def test_structured_output_tool(transport: MCPTransport, mcp_provider: MCPProvider): - """Test that tools with structured output (TypedDict) work correctly.""" - # Register the provider - await transport.register_tool_provider(mcp_provider) - - # Call the echo tool and verify the result - result = await transport.call_tool("echo", {"message": "test"}, mcp_provider) - assert result == {"reply": "you said: test"} - -@pytest.mark.asyncio -async def test_unstructured_string_output(transport: MCPTransport, mcp_provider: MCPProvider): - """Test that tools returning plain strings work correctly.""" - # Register the provider - await transport.register_tool_provider(mcp_provider) - - # Call the greet tool which returns a plain string - result = await transport.call_tool("greet", {"name": "Alice"}, mcp_provider) - assert result == "Hello, Alice!" - -@pytest.mark.asyncio -async def test_list_output(transport: MCPTransport, mcp_provider: MCPProvider): - """Test that tools returning lists work correctly.""" - # Register the provider - await transport.register_tool_provider(mcp_provider) - - # Call the list_items tool - result = await transport.call_tool("list_items", {"count": 3}, mcp_provider) - - # The result should be a list or wrapped in a result field - if isinstance(result, dict) and "result" in result: - items = result["result"] - else: - items = result - - assert isinstance(items, list) - assert len(items) == 3 - assert items == ["item_0", "item_1", "item_2"] - -@pytest.mark.asyncio -async def test_numeric_output(transport: MCPTransport, mcp_provider: MCPProvider): - """Test that tools returning numeric values work correctly.""" - # Register the provider - await transport.register_tool_provider(mcp_provider) - - # Call the add_numbers tool - result = await transport.call_tool("add_numbers", {"a": 5, "b": 7}, mcp_provider) - - # The result should be a number or wrapped in a result field - if isinstance(result, dict) and "result" in result: - value = result["result"] - else: - value = result - - assert value == 12 - -@pytest.mark.asyncio -async def test_deregister_provider(transport: MCPTransport, mcp_provider: MCPProvider): - """Verify that deregistering a provider works (no-op in session-per-operation mode).""" - # Register a provider - tools = await transport.register_tool_provider(mcp_provider) - assert len(tools) == 4 - - # Deregister it (this is a no-op in session-per-operation mode) - await transport.deregister_tool_provider(mcp_provider) - - # Should still be able to call tools since we create fresh sessions - result = await transport.call_tool("echo", {"message": "test"}, mcp_provider) - assert result == {"reply": "you said: test"} diff --git a/tests/client/transport_interfaces/test_sse_transport.py b/tests/client/transport_interfaces/test_sse_transport.py deleted file mode 100644 index 5c4ed31..0000000 --- a/tests/client/transport_interfaces/test_sse_transport.py +++ /dev/null @@ -1,320 +0,0 @@ -import pytest -import pytest_asyncio -import json -import asyncio -import base64 -from unittest.mock import MagicMock, patch, AsyncMock - -import aiohttp -from aiohttp import web - -from utcp.client.transport_interfaces.sse_transport import SSEClientTransport -from utcp.shared.provider import SSEProvider -from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth - -# --- Test Data --- - -SAMPLE_TOOLS_JSON = { - "version": "1.0", - "tools": [ - { - "name": "test_tool", - "description": "Test tool", - "inputs": { - "type": "object", - "properties": {"param1": {"type": "string"}} - }, - "outputs": { - "type": "object", - "properties": {"result": {"type": "string"}} - }, - "tags": [], - "tool_provider": { - "provider_type": "sse", - "name": "test-sse-provider-executor", - "url": "/events", - "http_method": "GET", - "content_type": "application/json" - } - } - ] -} - -SAMPLE_SSE_EVENTS = [ - 'id: 1\ndata: {"message": "First part"}\n\n', - 'id: 2\nevent: data\ndata: { "message": "Second part" }\n\n', - 'id: 3\nevent: complete\ndata: { "message": "End of stream" }\n\n' -] - -# --- Test Server Handlers --- - -async def tools_handler(request): - execution_provider = { - "provider_type": "sse", - "name": "test-sse-provider-executor", - "url": str(request.url.origin()) + "/events", - "http_method": "GET", - "content_type": "application/json" - } - utcp_manual = { - "version": "1.0", - "tools": [ - { - "name": "test_tool", - "description": "Test tool", - "inputs": { - "type": "object", - "properties": {"param1": {"type": "string"}} - }, - "outputs": { - "type": "object", - "properties": {"result": {"type": "string"}} - }, - "tags": [], - "tool_provider": execution_provider - } - ] - } - return web.json_response(utcp_manual) - -async def sse_handler(request): - # Check auth - if 'X-API-Key' in request.headers and request.headers['X-API-Key'] != 'test-api-key': - return web.Response(status=401, text="Invalid API Key") - if 'Authorization' in request.headers: - auth_header = request.headers['Authorization'] - if auth_header.startswith('Basic'): - # Basic dXNlcjpwYXNz - if auth_header != f"Basic {base64.b64encode(b'user:pass').decode()}": - return web.Response(status=401, text="Invalid Basic Auth") - elif auth_header.startswith('Bearer'): - if auth_header not in ('Bearer test-access-token', 'Bearer test-access-token-header'): - return web.Response(status=401, text="Invalid Bearer Token") - - response = web.StreamResponse( - status=200, - reason='OK', - headers={'Content-Type': 'text/event-stream'} - ) - await response.prepare(request) - - for event in SAMPLE_SSE_EVENTS: - await response.write(event.encode('utf-8')) - await asyncio.sleep(0.01) # Simulate network delay - - return response - -async def token_handler(request): - # OAuth2 token endpoint (credentials in body) - data = await request.post() - if data.get('client_id') == 'client-id' and data.get('client_secret') == 'client-secret': - return web.json_response({ - "access_token": "test-access-token", - "token_type": "Bearer", - "expires_in": 3600 - }) - return web.json_response({"error": "invalid_client"}, status=401) - -async def token_header_auth_handler(request): - # OAuth2 token endpoint (credentials in header) - auth_header = request.headers.get('Authorization') - if not auth_header or not auth_header.startswith('Basic '): - return web.json_response({"error": "missing_auth"}, status=401) - - return web.json_response({ - "access_token": "test-access-token-header", - "token_type": "Bearer", - "expires_in": 3600 - }) - -async def error_handler(request): - return web.Response(status=500, text="Internal Server Error") - -# --- Pytest Fixtures --- - -@pytest.fixture -def logger(): - return MagicMock() - -@pytest_asyncio.fixture -async def sse_transport(logger): - """Fixture to create and properly tear down an SSEClientTransport instance.""" - transport = SSEClientTransport(logger=logger) - yield transport - await transport.close() - -@pytest.fixture -def app(): - application = web.Application() - application.router.add_get("/tools", tools_handler) - application.router.add_get("/events", sse_handler) - application.router.add_post("/events", sse_handler) - application.router.add_post("/token", token_handler) - application.router.add_post("/token_header_auth", token_header_auth_handler) - application.router.add_get("/error", error_handler) - return application - -import pytest_asyncio - -@pytest_asyncio.fixture -async def oauth2_provider(aiohttp_client, app): - client = await aiohttp_client(app) - return SSEProvider( - name="oauth2-provider", - url=f"{client.make_url('/events')}", - auth=OAuth2Auth( - client_id="client-id", - client_secret="client-secret", - token_url=f"{client.make_url('/token')}" - ) - ) - -# --- Tests --- - -@pytest.mark.asyncio -async def test_register_tool_provider(sse_transport, aiohttp_client, app): - """Test registering a tool provider.""" - client = await aiohttp_client(app) - provider = SSEProvider(name="test", url=f"{client.make_url('/tools')}") - tools = await sse_transport.register_tool_provider(provider) - assert len(tools) == 1 - assert tools[0].name == "test_tool" - -@pytest.mark.asyncio -async def test_register_tool_provider_error(sse_transport, aiohttp_client, app, logger): - """Test error handling when registering a tool provider.""" - client = await aiohttp_client(app) - provider = SSEProvider(name="test-error", url=f"{client.make_url('/error')}") - tools = await sse_transport.register_tool_provider(provider) - # Only verify that the function returns an empty list of tools when an error occurs - assert tools == [] - -@pytest.mark.asyncio -async def test_call_tool_basic(sse_transport, aiohttp_client, app): - """Test calling a tool with basic configuration.""" - client = await aiohttp_client(app) - provider = SSEProvider(name="test-basic", url=f"{client.make_url('/events')}") - - stream_iterator = await sse_transport.call_tool("test_tool", {"param1": "value1"}, provider) - - results = [] - async for event in stream_iterator: - results.append(event) - - assert len(results) == 3 - assert results[0] == {"message": "First part"} - assert results[1] == {"message": "Second part"} - -@pytest.mark.asyncio -async def test_call_tool_with_api_key(sse_transport, aiohttp_client, app): - """Test calling a tool with API key authentication.""" - client = await aiohttp_client(app) - provider = SSEProvider( - name="api-key-provider", - url=f"{client.make_url('/events')}", - auth=ApiKeyAuth(var_name="X-API-Key", api_key="test-api-key") - ) - stream_iterator = await sse_transport.call_tool("test_tool", {}, provider) - results = [event async for event in stream_iterator] - assert len(results) == 3 - -@pytest.mark.asyncio -async def test_call_tool_with_basic_auth(sse_transport, aiohttp_client, app): - """Test calling a tool with Basic authentication.""" - client = await aiohttp_client(app) - provider = SSEProvider( - name="basic-auth-provider", - url=f"{client.make_url('/events')}", - auth=BasicAuth(username="user", password="pass") - ) - stream_iterator = await sse_transport.call_tool("test_tool", {}, provider) - results = [event async for event in stream_iterator] - assert len(results) == 3 - -@pytest.mark.asyncio -async def test_call_tool_with_oauth2(sse_transport, oauth2_provider, app): - """Test calling a tool with OAuth2 authentication (credentials in body).""" - # The provider fixture is already configured with the correct client URL - events = [] - async for event in await sse_transport.call_tool("test_tool", {"param1": "value1"}, oauth2_provider): - events.append(event) - - assert len(events) == 3 - assert events[0] == {"message": "First part"} - assert events[1] == {"message": "Second part"} - assert events[2] == {"message": "End of stream"} - -@pytest.mark.asyncio -async def test_call_tool_with_oauth2_header_auth(sse_transport, aiohttp_client, app): - """Test calling a tool with OAuth2 authentication (credentials in header).""" - client = await aiohttp_client(app) - oauth2_header_provider = SSEProvider( - name="oauth2-header-provider", - url=f"{client.make_url('/events')}", - auth=OAuth2Auth( - client_id="client-id", - client_secret="client-secret", - token_url=f"{client.make_url('/token_header_auth')}", - scope="read write" - ) - ) - - events = [] - async for event in await sse_transport.call_tool("test_tool", {"param1": "value1"}, oauth2_header_provider): - events.append(event) - - assert len(events) == 3 - assert events[0] == {"message": "First part"} - assert events[1] == {"message": "Second part"} - assert events[2] == {"message": "End of stream"} - -@pytest.mark.asyncio -async def test_call_tool_with_body_field(sse_transport, aiohttp_client, app): - """Test calling a tool with a body field.""" - client = await aiohttp_client(app) - provider = SSEProvider( - name="body-field-provider", - url=f"{client.make_url('/events')}", - body_field="data", - headers={"Content-Type": "application/json"} - ) - stream_iterator = await sse_transport.call_tool( - "test_tool", - {"param1": "value1", "data": {"key": "value"}}, - provider - ) - results = [event async for event in stream_iterator] - assert len(results) == 3 - -@pytest.mark.asyncio -async def test_call_tool_error(sse_transport, aiohttp_client, app, logger): - """Test error handling when calling a tool.""" - client = await aiohttp_client(app) - provider = SSEProvider(name="test-error", url=f"{client.make_url('/error')}") - with pytest.raises(aiohttp.ClientResponseError) as excinfo: - await sse_transport.call_tool("test_tool", {}, provider) - - assert excinfo.value.status == 500 - logger.assert_called_with(f"Error establishing SSE connection to '{provider.name}': 500, message='Internal Server Error', url='{provider.url}'", error=True) - -@pytest.mark.asyncio -async def test_deregister_tool_provider(sse_transport, aiohttp_client, app): - """Test deregistering a tool provider closes the connection.""" - client = await aiohttp_client(app) - provider = SSEProvider(name="test-deregister", url=f"{client.make_url('/events')}") - - # Make a call to establish a connection - stream_iterator = await sse_transport.call_tool("test_tool", {}, provider) - assert provider.name in sse_transport._active_connections - response, session = sse_transport._active_connections[provider.name] - - # Consume one item to ensure connection is active - await anext(stream_iterator) - - # Deregister - await sse_transport.deregister_tool_provider(provider) - - # Verify connection and session are closed and removed - assert provider.name not in sse_transport._active_connections - assert response.closed - assert session.closed diff --git a/tests/client/transport_interfaces/test_streamable_http_transport.py b/tests/client/transport_interfaces/test_streamable_http_transport.py deleted file mode 100644 index 3e11809..0000000 --- a/tests/client/transport_interfaces/test_streamable_http_transport.py +++ /dev/null @@ -1,245 +0,0 @@ -import pytest -import pytest_asyncio -import json -import asyncio -from unittest.mock import MagicMock - -from aiohttp import web - -from utcp.client.transport_interfaces.streamable_http_transport import StreamableHttpClientTransport -from utcp.shared.provider import StreamableHttpProvider -from utcp.shared.auth import ApiKeyAuth, BasicAuth, OAuth2Auth - -# --- Test Data --- - -SAMPLE_TOOLS_JSON = { - "version": "1.0", - "tools": [ - { - "name": "test_tool", - "description": "Test tool", - "inputs": {}, - "outputs": {}, - "tags": [], - "tool_provider": { - "provider_type": "http_stream", - "name": "test-streamable-http-provider-executor", - "url": "http://test-url/tool", - "http_method": "GET", - "content_type": "application/json" - } - } - ] -} - -SAMPLE_NDJSON_RESPONSE = [ - {'status': 'running', 'progress': 0}, - {'status': 'running', 'progress': 50}, - {'status': 'completed', 'result': 'done'} -] - -# --- Fixtures --- - -@pytest.fixture -def logger(): - """Fixture for a mock logger.""" - return MagicMock() - -@pytest_asyncio.fixture -async def streamable_http_transport(logger): - """Fixture to create and properly tear down a StreamableHttpClientTransport instance.""" - transport = StreamableHttpClientTransport(logger=logger) - yield transport - await transport.close() - -@pytest.fixture -def app(): - """Fixture for the aiohttp test application.""" - async def discover(request): - execution_provider = { - "provider_type": "http_stream", - "name": "test-streamable-http-provider-executor", - "url": str(request.url.origin()) + "/stream-ndjson", - "http_method": "GET", - "content_type": "application/x-ndjson" - } - utcp_manual = { - "version": "1.0", - "tools": [ - { - "name": "test_tool", - "description": "Test tool", - "inputs": {}, - "outputs": {}, - "tags": [], - "tool_provider": execution_provider - } - ] - } - return web.json_response(utcp_manual) - - async def stream_ndjson(request): - response = web.StreamResponse( - status=200, - reason='OK', - headers={'Content-Type': 'application/x-ndjson'} - ) - await response.prepare(request) - for item in SAMPLE_NDJSON_RESPONSE: - await response.write(json.dumps(item).encode('utf-8') + b'\n') - await asyncio.sleep(0.01) # Simulate network delay - await response.write_eof() - return response - - async def stream_binary(request): - response = web.StreamResponse( - status=200, - reason='OK', - headers={'Content-Type': 'application/octet-stream'} - ) - await response.prepare(request) - await response.write(b'chunk1') - await response.write(b'chunk2') - await response.write_eof() - return response - - async def check_api_key_auth(request): - if request.headers.get("X-API-Key") != "test-key": - return web.Response(status=401, text="Unauthorized: Invalid API Key") - return await stream_ndjson(request) - - async def check_basic_auth(request): - auth_header = request.headers.get('Authorization') - if not auth_header or 'Basic dXNlcjpwYXNz' not in auth_header: # user:pass - return web.Response(status=401, text="Unauthorized: Invalid Basic Auth") - return await stream_ndjson(request) - - async def oauth_token_handler(request): - data = await request.post() - if data.get('client_id') == 'test-client' and data.get('client_secret') == 'test-secret': - return web.json_response({'access_token': 'token-from-body', 'token_type': 'Bearer'}) - return web.Response(status=401, text="Invalid client credentials") - - async def oauth_token_header_handler(request): - auth_header = request.headers.get('Authorization') - if auth_header and 'Basic dGVzdC1jbGllbnQ6dGVzdC1zZWNyZXQ=' in auth_header: # test-client:test-secret - return web.json_response({'access_token': 'token-from-header', 'token_type': 'Bearer'}) - return web.Response(status=401, text="Invalid client credentials via header") - - async def check_oauth(request): - auth_header = request.headers.get('Authorization') - if auth_header in ('Bearer token-from-body', 'Bearer token-from-header'): - return await stream_ndjson(request) - return web.Response(status=401, text="Unauthorized: Invalid OAuth Token") - - async def error_endpoint(request): - return web.Response(status=500, text="Internal Server Error") - - app = web.Application() - app.add_routes([ - web.get('/discover', discover), - web.get('/stream-ndjson', stream_ndjson), - web.get('/stream-binary', stream_binary), - web.get('/auth-api-key', check_api_key_auth), - web.get('/auth-basic', check_basic_auth), - web.get('/auth-oauth', check_oauth), - web.post('/token', oauth_token_handler), - web.post('/token-header', oauth_token_header_handler), - web.get('/error', error_endpoint), - ]) - return app - -# --- Test Cases --- - -@pytest.mark.asyncio -async def test_register_tool_provider(streamable_http_transport, aiohttp_client, app): - """Test successful tool provider registration.""" - client = await aiohttp_client(app) - provider = StreamableHttpProvider(name="test-provider", url=f"{client.make_url('/discover')}") - tools = await streamable_http_transport.register_tool_provider(provider) - assert len(tools) == 1 - assert tools[0].name == "test_tool" - -@pytest.mark.asyncio -async def test_register_tool_provider_error(streamable_http_transport, aiohttp_client, app, logger): - """Test error handling during tool provider registration.""" - client = await aiohttp_client(app) - provider = StreamableHttpProvider(name="test-provider", url=f"{client.make_url('/error')}") - tools = await streamable_http_transport.register_tool_provider(provider) - assert tools == [] - assert logger.call_count > 0 - log_message = logger.call_args[0][0] - assert "Error discovering tools" in log_message - -@pytest.mark.asyncio -async def test_call_tool_ndjson_stream(streamable_http_transport, aiohttp_client, app): - """Test calling a tool that returns an NDJSON stream.""" - client = await aiohttp_client(app) - provider = StreamableHttpProvider(name="ndjson-provider", url=f"{client.make_url('/stream-ndjson')}", content_type='application/x-ndjson') - - stream_iterator = await streamable_http_transport.call_tool("test_tool", {}, provider) - - results = [item async for item in stream_iterator] - - assert results == SAMPLE_NDJSON_RESPONSE - -@pytest.mark.asyncio -async def test_call_tool_binary_stream(streamable_http_transport, aiohttp_client, app): - """Test calling a tool that returns a binary stream.""" - client = await aiohttp_client(app) - provider = StreamableHttpProvider(name="binary-provider", url=f"{client.make_url('/stream-binary')}", content_type='application/octet-stream', chunk_size=6) - - stream_iterator = await streamable_http_transport.call_tool("test_tool", {}, provider) - - results = [chunk async for chunk in stream_iterator] - - assert results == [b'chunk1', b'chunk2'] - -@pytest.mark.asyncio -async def test_call_tool_with_api_key(streamable_http_transport, aiohttp_client, app): - """Test that the API key is correctly sent in the headers.""" - client = await aiohttp_client(app) - auth = ApiKeyAuth(var_name="X-API-Key", api_key="test-key") - provider = StreamableHttpProvider(name="auth-provider", url=f"{client.make_url('/auth-api-key')}", auth=auth, content_type='application/x-ndjson') - - stream_iterator = await streamable_http_transport.call_tool("test_tool", {}, provider) - results = [item async for item in stream_iterator] - - assert results == SAMPLE_NDJSON_RESPONSE - -@pytest.mark.asyncio -async def test_call_tool_with_basic_auth(streamable_http_transport, aiohttp_client, app): - """Test streaming with Basic authentication.""" - client = await aiohttp_client(app) - auth = BasicAuth(username="user", password="pass") - provider = StreamableHttpProvider(name="basic-auth-provider", url=f"{client.make_url('/auth-basic')}", auth=auth, content_type='application/x-ndjson') - - stream_iterator = await streamable_http_transport.call_tool("test_tool", {}, provider) - results = [item async for item in stream_iterator] - - assert results == SAMPLE_NDJSON_RESPONSE - -@pytest.mark.asyncio -async def test_call_tool_with_oauth2_body(streamable_http_transport, aiohttp_client, app): - """Test streaming with OAuth2 (credentials in body).""" - client = await aiohttp_client(app) - auth = OAuth2Auth(client_id="test-client", client_secret="test-secret", token_url=f"{client.make_url('/token')}") - provider = StreamableHttpProvider(name="oauth-provider", url=f"{client.make_url('/auth-oauth')}", auth=auth, content_type='application/x-ndjson') - - stream_iterator = await streamable_http_transport.call_tool("test_tool", {}, provider) - results = [item async for item in stream_iterator] - - assert results == SAMPLE_NDJSON_RESPONSE - -@pytest.mark.asyncio -async def test_call_tool_with_oauth2_header_fallback(streamable_http_transport, aiohttp_client, app): - """Test streaming with OAuth2 (fallback to Basic Auth header).""" - client = await aiohttp_client(app) - # This token endpoint will fail for the body method, forcing a fallback. - auth = OAuth2Auth(client_id="test-client", client_secret="test-secret", token_url=f"{client.make_url('/token-header')}") - provider = StreamableHttpProvider(name="oauth-fallback-provider", url=f"{client.make_url('/auth-oauth')}", auth=auth, content_type='application/x-ndjson') - - stream_iterator = await streamable_http_transport.call_tool("test_tool", {}, provider) - results = [item async for item in stream_iterator] - - assert results == SAMPLE_NDJSON_RESPONSE diff --git a/tests/client/transport_interfaces/test_tcp_transport.py b/tests/client/transport_interfaces/test_tcp_transport.py deleted file mode 100644 index c32dcdf..0000000 --- a/tests/client/transport_interfaces/test_tcp_transport.py +++ /dev/null @@ -1,875 +0,0 @@ -import pytest -import pytest_asyncio -import json -import asyncio -import socket -import struct -import threading -from unittest.mock import MagicMock, patch, AsyncMock - -from utcp.client.transport_interfaces.tcp_transport import TCPTransport -from utcp.shared.provider import TCPProvider -from utcp.shared.tool import Tool, ToolInputOutputSchema - - -class MockTCPServer: - """Mock TCP server for testing.""" - - def __init__(self, host='localhost', port=0, response_delay=0.0): - self.host = host - self.port = port - self.sock = None - self.running = False - self.responses = {} # Map message -> response - self.call_count = 0 - self.server_task = None - self.connections = [] - self.response_delay = response_delay # Delay before sending response (seconds) - - async def start(self): - """Start the mock TCP server.""" - # Create socket and bind - self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) - self.sock.bind((self.host, self.port)) - if self.port == 0: # Auto-assign port - self.port = self.sock.getsockname()[1] - - self.sock.listen(5) - self.running = True - - # Start listening task - self.server_task = asyncio.create_task(self._accept_connections()) - - # Give the server a moment to start - await asyncio.sleep(0.1) - - async def stop(self): - """Stop the mock TCP server.""" - self.running = False - if self.server_task: - self.server_task.cancel() - try: - await self.server_task - except asyncio.CancelledError: - pass - - # Close all active connections - for conn in self.connections: - try: - conn.close() - except Exception: - pass - self.connections.clear() - - if self.sock: - self.sock.close() - - async def _accept_connections(self): - """Accept incoming TCP connections.""" - self.sock.setblocking(False) - - while self.running: - try: - conn, addr = await asyncio.get_event_loop().sock_accept(self.sock) - self.connections.append(conn) - # Handle each connection in a separate task - asyncio.create_task(self._handle_connection(conn, addr)) - except asyncio.CancelledError: - break - except Exception as e: - if self.running: - print(f"Mock TCP server accept error: {e}") - await asyncio.sleep(0.01) - - async def _handle_connection(self, conn, addr): - """Handle a single TCP connection.""" - try: - # Read data from client - data = await asyncio.get_event_loop().sock_recv(conn, 4096) - if not data: - return - - self.call_count += 1 - - try: - message = data.decode('utf-8') - except UnicodeDecodeError: - message = data.hex() # Fallback for binary data - - # Get response for this message - response = self.responses.get(message, '{"error": "unknown_message"}') - - # Convert response to bytes - if isinstance(response, str): - response_bytes = response.encode('utf-8') - elif isinstance(response, bytes): - response_bytes = response - elif isinstance(response, dict) or isinstance(response, list): - response_bytes = json.dumps(response).encode('utf-8') - else: - response_bytes = str(response).encode('utf-8') - - # Add delay if configured - if self.response_delay > 0: - await asyncio.sleep(self.response_delay) - - # Send response back - await asyncio.get_event_loop().sock_sendall(conn, response_bytes) - - except Exception as e: - if self.running: - print(f"Mock TCP server connection error: {e}") - finally: - try: - conn.close() - except Exception: - pass - if conn in self.connections: - self.connections.remove(conn) - - def set_response(self, message, response): - """Set a response for a specific message.""" - self.responses[message] = response - - -class MockTCPServerWithFraming(MockTCPServer): - """Mock TCP server that handles different framing strategies.""" - - def __init__(self, host='localhost', port=0, framing_strategy='stream', response_delay=0.0): - super().__init__(host, port, response_delay) - self.framing_strategy = framing_strategy - self.length_prefix_bytes = 4 - self.length_prefix_endian = 'big' - self.message_delimiter = '\n' - self.fixed_message_length = None - - async def _handle_connection(self, conn, addr): - """Handle a single TCP connection with framing.""" - try: - if self.framing_strategy == 'length_prefix': - # Read length prefix first - length_data = await asyncio.get_event_loop().sock_recv(conn, self.length_prefix_bytes) - if not length_data: - return - - if self.length_prefix_bytes == 1: - message_length = struct.unpack(f"{'>' if self.length_prefix_endian == 'big' else '<'}B", length_data)[0] - elif self.length_prefix_bytes == 2: - message_length = struct.unpack(f"{'>' if self.length_prefix_endian == 'big' else '<'}H", length_data)[0] - elif self.length_prefix_bytes == 4: - message_length = struct.unpack(f"{'>' if self.length_prefix_endian == 'big' else '<'}I", length_data)[0] - - # Read the actual message - data = await asyncio.get_event_loop().sock_recv(conn, message_length) - - elif self.framing_strategy == 'delimiter': - # Read until delimiter - data = b'' - delimiter_bytes = self.message_delimiter.encode('utf-8') - while not data.endswith(delimiter_bytes): - chunk = await asyncio.get_event_loop().sock_recv(conn, 1) - if not chunk: - break - data += chunk - # Remove delimiter - data = data[:-len(delimiter_bytes)] - - elif self.framing_strategy == 'fixed_length': - # Read fixed number of bytes - data = await asyncio.get_event_loop().sock_recv(conn, self.fixed_message_length) - - else: # stream - # Read all available data - data = await asyncio.get_event_loop().sock_recv(conn, 4096) - - if not data: - return - - self.call_count += 1 - - try: - message = data.decode('utf-8') - except UnicodeDecodeError: - message = data.hex() - - # Get response for this message - response = self.responses.get(message, '{"error": "unknown_message"}') - - # Convert response to bytes - if isinstance(response, str): - response_bytes = response.encode('utf-8') - elif isinstance(response, bytes): - response_bytes = response - elif isinstance(response, dict) or isinstance(response, list): - response_bytes = json.dumps(response).encode('utf-8') - else: - response_bytes = str(response).encode('utf-8') - - # Add delay if configured - if self.response_delay > 0: - await asyncio.sleep(self.response_delay) - - # Send response with appropriate framing - if self.framing_strategy == 'length_prefix': - # Add length prefix - length = len(response_bytes) - if self.length_prefix_bytes == 1: - length_bytes = struct.pack(f"{'>' if self.length_prefix_endian == 'big' else '<'}B", length) - elif self.length_prefix_bytes == 2: - length_bytes = struct.pack(f"{'>' if self.length_prefix_endian == 'big' else '<'}H", length) - elif self.length_prefix_bytes == 4: - length_bytes = struct.pack(f"{'>' if self.length_prefix_endian == 'big' else '<'}I", length) - - await asyncio.get_event_loop().sock_sendall(conn, length_bytes + response_bytes) - - elif self.framing_strategy == 'delimiter': - # Add delimiter - delimiter_bytes = self.message_delimiter.encode('utf-8') - await asyncio.get_event_loop().sock_sendall(conn, response_bytes + delimiter_bytes) - - else: # stream or fixed_length - await asyncio.get_event_loop().sock_sendall(conn, response_bytes) - - except Exception as e: - if self.running: - print(f"Mock TCP server connection error: {e}") - finally: - try: - conn.close() - except Exception: - pass - if conn in self.connections: - self.connections.remove(conn) - - -@pytest_asyncio.fixture -async def mock_tcp_server(): - """Create a mock TCP server for testing.""" - server = MockTCPServer() - await server.start() - yield server - await server.stop() - - -@pytest_asyncio.fixture -async def mock_tcp_server_length_prefix(): - """Create a mock TCP server with length-prefix framing.""" - server = MockTCPServerWithFraming(framing_strategy='length_prefix') - await server.start() - yield server - await server.stop() - - -@pytest_asyncio.fixture -async def mock_tcp_server_delimiter(): - """Create a mock TCP server with delimiter framing.""" - server = MockTCPServerWithFraming(framing_strategy='delimiter') - await server.start() - yield server - await server.stop() - - -@pytest_asyncio.fixture -async def mock_tcp_server_slow(): - """Create a mock TCP server with a 2-second response delay.""" - server = MockTCPServer(response_delay=2.0) # 2-second delay - await server.start() - yield server - await server.stop() - - -@pytest.fixture -def logger(): - """Create a mock logger.""" - return MagicMock() - - -@pytest.fixture -def tcp_transport(logger): - """Create a TCP transport instance.""" - return TCPTransport(logger=logger) - - -@pytest.fixture -def tcp_provider(mock_tcp_server): - """Create a basic TCP provider for testing.""" - return TCPProvider( - name="test_tcp_provider", - host=mock_tcp_server.host, - port=mock_tcp_server.port, - request_data_format="json", - response_byte_format="utf-8", - framing_strategy="stream", - timeout=5000 - ) - - -@pytest.fixture -def text_template_provider(mock_tcp_server): - """Create a TCP provider with text template format.""" - return TCPProvider( - name="text_template_provider", - host=mock_tcp_server.host, - port=mock_tcp_server.port, - request_data_format="text", - request_data_template="ACTION UTCP_ARG_cmd_UTCP_ARG PARAM UTCP_ARG_value_UTCP_ARG", - response_byte_format="utf-8", - framing_strategy="stream", - timeout=5000 - ) - - -@pytest.fixture -def raw_bytes_provider(mock_tcp_server): - """Create a TCP provider that returns raw bytes.""" - return TCPProvider( - name="raw_bytes_provider", - host=mock_tcp_server.host, - port=mock_tcp_server.port, - request_data_format="json", - response_byte_format=None, # Raw bytes - framing_strategy="stream", - timeout=5000 - ) - - -@pytest.fixture -def length_prefix_provider(mock_tcp_server_length_prefix): - """Create a TCP provider with length-prefix framing.""" - return TCPProvider( - name="length_prefix_provider", - host=mock_tcp_server_length_prefix.host, - port=mock_tcp_server_length_prefix.port, - request_data_format="json", - response_byte_format="utf-8", - framing_strategy="length_prefix", - length_prefix_bytes=4, - length_prefix_endian="big", - timeout=5000 - ) - - -@pytest.fixture -def delimiter_provider(mock_tcp_server_delimiter): - """Create a TCP provider with delimiter framing.""" - return TCPProvider( - name="delimiter_provider", - host=mock_tcp_server_delimiter.host, - port=mock_tcp_server_delimiter.port, - request_data_format="json", - response_byte_format="utf-8", - framing_strategy="delimiter", - message_delimiter="\n", - timeout=5000 - ) - - -# Test register_tool_provider -@pytest.mark.asyncio -async def test_register_tool_provider(tcp_transport, tcp_provider, mock_tcp_server, logger): - """Test registering a tool provider.""" - # Set up discovery response - discovery_response = { - "tools": [ - { - "name": "test_tool", - "description": "A test tool", - "inputs": { - "type": "object", - "properties": { - "param1": {"type": "string", "description": "First parameter"} - }, - "required": ["param1"] - }, - "outputs": { - "type": "object", - "properties": { - "result": {"type": "string", "description": "Result"} - } - }, - "tool_provider": tcp_provider.model_dump() - } - ] - } - - mock_tcp_server.set_response('{"type": "utcp"}', discovery_response) - - # Register the provider - tools = await tcp_transport.register_tool_provider(tcp_provider) - - # Check results - assert len(tools) == 1 - assert tools[0].name == "test_tool" - assert tools[0].description == "A test tool" - assert mock_tcp_server.call_count == 1 - - # Verify logger was called - logger.assert_called() - - -@pytest.mark.asyncio -async def test_register_tool_provider_empty_response(tcp_transport, tcp_provider, mock_tcp_server): - """Test registering a tool provider with empty response.""" - mock_tcp_server.set_response('{"type": "utcp"}', {"tools": []}) - - tools = await tcp_transport.register_tool_provider(tcp_provider) - - assert len(tools) == 0 - assert mock_tcp_server.call_count == 1 - - -@pytest.mark.asyncio -async def test_register_tool_provider_invalid_json(tcp_transport, tcp_provider, mock_tcp_server): - """Test registering a tool provider with invalid JSON response.""" - mock_tcp_server.set_response('{"type": "utcp"}', "invalid json response") - - tools = await tcp_transport.register_tool_provider(tcp_provider) - - assert len(tools) == 0 - - -@pytest.mark.asyncio -async def test_register_tool_provider_invalid_provider_type(tcp_transport): - """Test registering a non-TCP provider raises ValueError.""" - from utcp.shared.provider import HttpProvider - - invalid_provider = HttpProvider(url="http://example.com") - - with pytest.raises(ValueError, match="TCPTransport can only be used with TCPProvider"): - await tcp_transport.register_tool_provider(invalid_provider) - - -# Test deregister_tool_provider -@pytest.mark.asyncio -async def test_deregister_tool_provider(tcp_transport, tcp_provider): - """Test deregistering a tool provider (should be a no-op).""" - # Should not raise any exceptions - await tcp_transport.deregister_tool_provider(tcp_provider) - - -@pytest.mark.asyncio -async def test_deregister_tool_provider_invalid_type(tcp_transport): - """Test deregistering a non-TCP provider raises ValueError.""" - from utcp.shared.provider import HttpProvider - - invalid_provider = HttpProvider(url="http://example.com") - - with pytest.raises(ValueError, match="TCPTransport can only be used with TCPProvider"): - await tcp_transport.deregister_tool_provider(invalid_provider) - - -# Test call_tool with JSON format -@pytest.mark.asyncio -async def test_call_tool_json_format(tcp_transport, tcp_provider, mock_tcp_server): - """Test calling a tool with JSON format.""" - mock_tcp_server.set_response('{"param1": "value1"}', '{"result": "success"}') - - arguments = {"param1": "value1"} - result = await tcp_transport.call_tool("test_tool", arguments, tcp_provider) - - assert result == '{"result": "success"}' - assert mock_tcp_server.call_count == 1 - - -@pytest.mark.asyncio -async def test_call_tool_text_template_format(tcp_transport, text_template_provider, mock_tcp_server): - """Test calling a tool with text template format.""" - mock_tcp_server.set_response("ACTION get PARAM data123", '{"result": "template_success"}') - - arguments = {"cmd": "get", "value": "data123"} - result = await tcp_transport.call_tool("test_tool", arguments, text_template_provider) - - assert result == '{"result": "template_success"}' - assert mock_tcp_server.call_count == 1 - - -@pytest.mark.asyncio -async def test_call_tool_text_format_no_template(tcp_transport, mock_tcp_server): - """Test calling a tool with text format but no template.""" - provider = TCPProvider( - name="no_template_provider", - host=mock_tcp_server.host, - port=mock_tcp_server.port, - request_data_format="text", - request_data_template=None, - response_byte_format="utf-8", - framing_strategy="stream", - timeout=5000 - ) - - # Should use fallback format (space-separated values) - mock_tcp_server.set_response("value1 value2", '{"result": "fallback_success"}') - - arguments = {"param1": "value1", "param2": "value2"} - result = await tcp_transport.call_tool("test_tool", arguments, provider) - - assert result == '{"result": "fallback_success"}' - - -@pytest.mark.asyncio -async def test_call_tool_raw_bytes_response(tcp_transport, raw_bytes_provider, mock_tcp_server): - """Test calling a tool that returns raw bytes.""" - binary_response = b'\x01\x02\x03\x04' - mock_tcp_server.set_response('{"param1": "value1"}', binary_response) - - arguments = {"param1": "value1"} - result = await tcp_transport.call_tool("test_tool", arguments, raw_bytes_provider) - - assert result == binary_response - assert isinstance(result, bytes) - - -@pytest.mark.asyncio -async def test_call_tool_invalid_provider_type(tcp_transport): - """Test calling a tool with non-TCP provider raises ValueError.""" - from utcp.shared.provider import HttpProvider - - invalid_provider = HttpProvider(url="http://example.com") - - with pytest.raises(ValueError, match="TCPTransport can only be used with TCPProvider"): - await tcp_transport.call_tool("test_tool", {}, invalid_provider) - - -# Test framing strategies -@pytest.mark.asyncio -async def test_call_tool_length_prefix_framing(tcp_transport, length_prefix_provider, mock_tcp_server_length_prefix): - """Test calling a tool with length-prefix framing.""" - mock_tcp_server_length_prefix.set_response('{"param1": "value1"}', '{"result": "length_prefix_success"}') - - arguments = {"param1": "value1"} - result = await tcp_transport.call_tool("test_tool", arguments, length_prefix_provider) - - assert result == '{"result": "length_prefix_success"}' - - -@pytest.mark.asyncio -async def test_call_tool_delimiter_framing(tcp_transport, delimiter_provider, mock_tcp_server_delimiter): - """Test calling a tool with delimiter framing.""" - mock_tcp_server_delimiter.set_response('{"param1": "value1"}', '{"result": "delimiter_success"}') - - arguments = {"param1": "value1"} - result = await tcp_transport.call_tool("test_tool", arguments, delimiter_provider) - - assert result == '{"result": "delimiter_success"}' - - -@pytest.mark.asyncio -async def test_call_tool_fixed_length_framing(tcp_transport, mock_tcp_server): - """Test calling a tool with fixed-length framing.""" - provider = TCPProvider( - name="fixed_length_provider", - host=mock_tcp_server.host, - port=mock_tcp_server.port, - request_data_format="json", - response_byte_format="utf-8", - framing_strategy="fixed_length", - fixed_message_length=20, - timeout=5000 - ) - - # Set up server to handle fixed-length messages - mock_tcp_server.responses['{"param1": "value1"}'] = '{"result": "fixed"}'.ljust(20) # Pad to 20 bytes - - arguments = {"param1": "value1"} - result = await tcp_transport.call_tool("test_tool", arguments, provider) - - assert '{"result": "fixed"}' in result - - -# Test message formatting -def test_format_tool_call_message_json(tcp_transport): - """Test formatting tool call message with JSON format.""" - provider = TCPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="json" - ) - - arguments = {"param1": "value1", "param2": 123} - result = tcp_transport._format_tool_call_message(arguments, provider) - - assert result == json.dumps(arguments) - - -def test_format_tool_call_message_text_with_template(tcp_transport): - """Test formatting tool call message with text template.""" - provider = TCPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="text", - request_data_template="ACTION UTCP_ARG_cmd_UTCP_ARG PARAM UTCP_ARG_value_UTCP_ARG" - ) - - arguments = {"cmd": "get", "value": "data123"} - result = tcp_transport._format_tool_call_message(arguments, provider) - - # Should substitute placeholders - assert result == "ACTION get PARAM data123" - - -def test_format_tool_call_message_text_with_complex_values(tcp_transport): - """Test formatting tool call message with complex values in template.""" - provider = TCPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="text", - request_data_template="DATA UTCP_ARG_obj_UTCP_ARG" - ) - - arguments = {"obj": {"nested": "value", "number": 123}} - result = tcp_transport._format_tool_call_message(arguments, provider) - - # Should JSON-serialize complex values - assert result == 'DATA {"nested": "value", "number": 123}' - - -def test_format_tool_call_message_text_no_template(tcp_transport): - """Test formatting tool call message with text format but no template.""" - provider = TCPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="text", - request_data_template=None - ) - - arguments = {"param1": "value1", "param2": "value2"} - result = tcp_transport._format_tool_call_message(arguments, provider) - - # Should use fallback format (space-separated values) - assert result == "value1 value2" - - -def test_format_tool_call_message_default_to_json(tcp_transport): - """Test formatting tool call message defaults to JSON for unknown format.""" - # Create a provider with valid format first - provider = TCPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="json" - ) - - # Manually set an invalid format to test the fallback behavior - provider.request_data_format = "unknown" # Invalid format - - arguments = {"param1": "value1"} - result = tcp_transport._format_tool_call_message(arguments, provider) - - # Should default to JSON - assert result == json.dumps(arguments) - - -# Test framing encoding and decoding -def test_encode_message_with_length_prefix_framing(tcp_transport): - """Test encoding message with length-prefix framing.""" - provider = TCPProvider( - name="test", - host="localhost", - port=1234, - framing_strategy="length_prefix", - length_prefix_bytes=4, - length_prefix_endian="big" - ) - - message = "test message" - result = tcp_transport._encode_message_with_framing(message, provider) - - # Should have 4-byte big-endian length prefix - expected_length = len(message.encode('utf-8')) - expected_prefix = struct.pack('>I', expected_length) - - assert result.startswith(expected_prefix) - assert result[4:] == message.encode('utf-8') - - -def test_encode_message_with_delimiter_framing(tcp_transport): - """Test encoding message with delimiter framing.""" - provider = TCPProvider( - name="test", - host="localhost", - port=1234, - framing_strategy="delimiter", - message_delimiter="\n" - ) - - message = "test message" - result = tcp_transport._encode_message_with_framing(message, provider) - - # Should have delimiter appended - assert result == (message + "\n").encode('utf-8') - - -def test_encode_message_with_stream_framing(tcp_transport): - """Test encoding message with stream framing.""" - provider = TCPProvider( - name="test", - host="localhost", - port=1234, - framing_strategy="stream" - ) - - message = "test message" - result = tcp_transport._encode_message_with_framing(message, provider) - - # Should just be the raw message - assert result == message.encode('utf-8') - - -# Test error handling and edge cases -@pytest.mark.asyncio -async def test_call_tool_server_error(tcp_transport, tcp_provider, mock_tcp_server): - """Test handling server errors during tool calls.""" - # Don't set any response, so the server will return an error - arguments = {"param1": "value1"} - - # Call the tool - should get the default error response - result = await tcp_transport.call_tool("test_tool", arguments, tcp_provider) - - # Should receive the default error message - assert '{"error": "unknown_message"}' in result - - -@pytest.mark.asyncio -async def test_register_tool_provider_malformed_tool(tcp_transport, tcp_provider, mock_tcp_server): - """Test registering provider with malformed tool definition.""" - # Set up discovery response with invalid tool - discovery_response = { - "tools": [ - { - "name": "test_tool", - # Missing required fields like inputs, outputs, tool_provider - } - ] - } - - mock_tcp_server.set_response('{"type": "utcp"}', discovery_response) - - # Register the provider - should handle invalid tool gracefully - tools = await tcp_transport.register_tool_provider(tcp_provider) - - # Should return empty list due to invalid tool definition - assert len(tools) == 0 - - -@pytest.mark.asyncio -async def test_register_tool_provider_bytes_response(tcp_transport, tcp_provider, mock_tcp_server): - """Test registering provider that returns bytes response.""" - # Set up discovery response as JSON but provider returns raw bytes - discovery_response = '{"tools": []}'.encode('utf-8') - - mock_tcp_server.set_response('{"type": "utcp"}', discovery_response) - - # Register the provider - should handle bytes response by decoding - tools = await tcp_transport.register_tool_provider(tcp_provider) - - # Should successfully decode and parse - assert len(tools) == 0 - - -# Test logging functionality -@pytest.mark.asyncio -async def test_logging_calls(tcp_transport, tcp_provider, mock_tcp_server, logger): - """Test that logging functions are called appropriately.""" - # Set up discovery response - discovery_response = {"tools": []} - mock_tcp_server.set_response('{"type": "utcp"}', discovery_response) - - # Register provider - await tcp_transport.register_tool_provider(tcp_provider) - - # Verify logger was called - logger.assert_called() - - # Call tool - mock_tcp_server.set_response('{}', {"result": "test"}) - await tcp_transport.call_tool("test_tool", {}, tcp_provider) - - # Logger should have been called multiple times - assert logger.call_count > 1 - - -# Test timeout handling -@pytest.mark.asyncio -async def test_call_tool_timeout(tcp_transport): - """Test calling a tool with timeout using delimiter framing.""" - # Create a slow server with delimiter framing - slow_server = MockTCPServerWithFraming( - framing_strategy='delimiter', - response_delay=2.0 # 2-second delay - ) - await slow_server.start() - - try: - # Create provider with 1-second timeout, but server has 2-second delay - provider = TCPProvider( - name="timeout_provider", - host=slow_server.host, - port=slow_server.port, - request_data_format="json", - response_byte_format="utf-8", - framing_strategy="delimiter", - message_delimiter="\n", - timeout=1000 # 1 second timeout, but server delays 2 seconds - ) - - # Set up a response (server will delay 2 seconds before responding) - slow_server.set_response('{"param1": "value1"}', '{"result": "delayed_response"}') - - arguments = {"param1": "value1"} - - # Should timeout because server takes 2 seconds but timeout is 1 second - # Delimiter framing will treat timeout as an error since it expects a complete message - with pytest.raises(Exception): # Expect timeout error - await tcp_transport.call_tool("test_tool", arguments, provider) - finally: - await slow_server.stop() - - -@pytest.mark.asyncio -async def test_call_tool_connection_refused(tcp_transport): - """Test calling a tool when connection is refused.""" - # Use a port that's definitely not listening - provider = TCPProvider( - name="refused_provider", - host="localhost", - port=1, # Port 1 should be refused - request_data_format="json", - response_byte_format="utf-8", - framing_strategy="stream", - timeout=5000 - ) - - arguments = {"param1": "value1"} - - # Should handle connection error gracefully - with pytest.raises(Exception): # Expect connection refused or similar - await tcp_transport.call_tool("test_tool", arguments, provider) - - -# Test different byte encodings -@pytest.mark.asyncio -async def test_call_tool_different_encodings(tcp_transport, mock_tcp_server): - """Test calling a tool with different response byte encodings.""" - # Test ASCII encoding - provider_ascii = TCPProvider( - name="ascii_provider", - host=mock_tcp_server.host, - port=mock_tcp_server.port, - request_data_format="json", - response_byte_format="ascii", - framing_strategy="stream", - timeout=5000 - ) - - mock_tcp_server.set_response('{"param1": "value1"}', '{"result": "ascii_success"}') - - arguments = {"param1": "value1"} - result = await tcp_transport.call_tool("test_tool", arguments, provider_ascii) - - assert result == '{"result": "ascii_success"}' - assert isinstance(result, str) diff --git a/tests/client/transport_interfaces/test_text_transport.py b/tests/client/transport_interfaces/test_text_transport.py deleted file mode 100644 index 0f67177..0000000 --- a/tests/client/transport_interfaces/test_text_transport.py +++ /dev/null @@ -1,360 +0,0 @@ -""" -Tests for the text file transport interface. -""" -import json -import tempfile -from pathlib import Path -import pytest -import pytest_asyncio - -from utcp.client.transport_interfaces.text_transport import TextTransport -from utcp.shared.provider import TextProvider - - -@pytest_asyncio.fixture -async def transport() -> TextTransport: - """Provides a clean TextTransport instance.""" - t = TextTransport() - yield t - await t.close() - - -@pytest_asyncio.fixture -def sample_utcp_manual(): - """Sample UTCP manual with multiple tools.""" - return { - "version": "1.0.0", - "name": "Sample Tools", - "description": "A collection of sample tools for testing", - "tools": [ - { - "name": "calculator", - "description": "Performs basic arithmetic operations", - "inputs": { - "properties": { - "operation": { - "type": "string", - "enum": ["add", "subtract", "multiply", "divide"] - }, - "a": {"type": "number"}, - "b": {"type": "number"} - }, - "required": ["operation", "a", "b"] - }, - "outputs": { - "properties": { - "result": {"type": "number"} - } - }, - "tags": ["math", "arithmetic"], - "tool_provider": { - "provider_type": "text", - "name": "test-text-provider", - "file_path": "dummy.json" - } - }, - { - "name": "string_utils", - "description": "String manipulation utilities", - "inputs": { - "properties": { - "text": {"type": "string"}, - "operation": { - "type": "string", - "enum": ["uppercase", "lowercase", "reverse"] - } - }, - "required": ["text", "operation"] - }, - "outputs": { - "properties": { - "result": {"type": "string"} - } - }, - "tags": ["text", "utilities"], - "tool_provider": { - "provider_type": "text", - "name": "test-text-provider", - "file_path": "dummy.json" - } - } - ] - } - - -@pytest_asyncio.fixture -def single_tool_definition(): - """Sample single tool definition.""" - return { - "name": "echo", - "description": "Echoes back the input text", - "inputs": { - "properties": { - "message": {"type": "string"} - }, - "required": ["message"] - }, - "outputs": { - "properties": { - "echo": {"type": "string"} - } - }, - "tags": ["utility"], - "tool_provider": { - "provider_type": "text", - "name": "test-text-provider", - "file_path": "dummy.json" - } - } - - -@pytest_asyncio.fixture -def tool_array(): - """Sample array of tool definitions.""" - return [ - { - "name": "tool1", - "description": "First tool", - "inputs": {"properties": {}, "required": []}, - "outputs": {"properties": {}, "required": []}, - "tags": [], - "tool_provider": { - "provider_type": "text", - "name": "test-text-provider", - "file_path": "dummy.json" - } - }, - { - "name": "tool2", - "description": "Second tool", - "inputs": {"properties": {}, "required": []}, - "outputs": {"properties": {}, "required": []}, - "tags": [], - "tool_provider": { - "provider_type": "text", - "name": "test-text-provider", - "file_path": "dummy.json" - } - } - ] - - -@pytest.mark.asyncio -async def test_register_provider_with_utcp_manual(transport: TextTransport, sample_utcp_manual): - """Test registering a provider with a UTCP manual format file.""" - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump(sample_utcp_manual, f) - temp_file = f.name - - try: - provider = TextProvider( - name="test_provider", - file_path=temp_file - ) - - tools = await transport.register_tool_provider(provider) - - assert len(tools) == 2 - assert tools[0].name == "calculator" - assert tools[0].description == "Performs basic arithmetic operations" - assert tools[0].tags == ["math", "arithmetic"] - assert tools[0].tool_provider.name == "test-text-provider" - - assert tools[1].name == "string_utils" - assert tools[1].description == "String manipulation utilities" - assert tools[1].tags == ["text", "utilities"] - assert tools[1].tool_provider.name == "test-text-provider" - - finally: - Path(temp_file).unlink() - - -@pytest.mark.asyncio -async def test_register_provider_with_single_tool(transport: TextTransport, single_tool_definition): - """Test registering a provider with a single tool definition.""" - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - manual = { - "version": "1.0.0", - "name": "Single Tool Manual", - "description": "A manual with a single tool", - "tools": [single_tool_definition] - } - json.dump(manual, f) - temp_file = f.name - - try: - provider = TextProvider( - name="single_tool_provider", - file_path=temp_file - ) - - tools = await transport.register_tool_provider(provider) - - assert len(tools) == 1 - assert tools[0].name == "echo" - assert tools[0].description == "Echoes back the input text" - assert tools[0].tags == ["utility"] - assert tools[0].tool_provider.name == "test-text-provider" - - finally: - Path(temp_file).unlink() - - -@pytest.mark.asyncio -async def test_register_provider_with_tool_array(transport: TextTransport, tool_array): - """Test registering a provider with an array of tool definitions.""" - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - manual = { - "version": "1.0.0", - "name": "Tool Array Manual", - "description": "A manual with a tool array", - "tools": tool_array - } - json.dump(manual, f) - temp_file = f.name - - try: - provider = TextProvider( - name="array_provider", - file_path=temp_file - ) - - tools = await transport.register_tool_provider(provider) - - assert len(tools) == 2 - assert tools[0].name == "tool1" - assert tools[1].name == "tool2" - assert tools[0].tool_provider.name == "test-text-provider" - assert tools[1].tool_provider.name == "test-text-provider" - - finally: - Path(temp_file).unlink() - - -@pytest.mark.asyncio -async def test_register_provider_file_not_found(transport: TextTransport): - """Test registering a provider with a non-existent file.""" - provider = TextProvider( - name="missing_file_provider", - file_path="/path/that/does/not/exist.json" - ) - - with pytest.raises(FileNotFoundError): - await transport.register_tool_provider(provider) - - -@pytest.mark.asyncio -async def test_register_provider_invalid_json(transport: TextTransport): - """Test registering a provider with invalid JSON.""" - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - f.write("{ invalid json content }") - temp_file = f.name - - try: - provider = TextProvider( - name="invalid_json_provider", - file_path=temp_file - ) - - with pytest.raises(json.JSONDecodeError): - await transport.register_tool_provider(provider) - - finally: - Path(temp_file).unlink() - - -@pytest.mark.asyncio -async def test_register_provider_wrong_type(transport: TextTransport): - """Test registering a provider with wrong provider type.""" - from utcp.shared.provider import HttpProvider - - provider = HttpProvider( - name="http_provider", - url="https://example.com" - ) - - with pytest.raises(ValueError, match="TextTransport can only be used with TextProvider"): - await transport.register_tool_provider(provider) - - -@pytest.mark.asyncio -async def test_call_tool_returns_file_content(transport: TextTransport, sample_utcp_manual): - """Test that calling tools returns the content of the text file.""" - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump(sample_utcp_manual, f) - temp_file = f.name - - try: - provider = TextProvider( - name="test_provider", - file_path=temp_file - ) - - # Register the provider first - await transport.register_tool_provider(provider) - - # Call a tool should return the file content - content = await transport.call_tool("calculator", {"operation": "add", "a": 1, "b": 2}, provider) - - # Verify we get the JSON content back as a string - assert isinstance(content, str) - # Parse it back to verify it's the same content - parsed_content = json.loads(content) - assert parsed_content == sample_utcp_manual - - finally: - Path(temp_file).unlink() - - -@pytest.mark.asyncio -async def test_call_tool_wrong_provider_type(transport: TextTransport): - """Test calling a tool with wrong provider type.""" - from utcp.shared.provider import HttpProvider - - provider = HttpProvider( - name="http_provider", - url="https://example.com" - ) - - with pytest.raises(ValueError, match="TextTransport can only be used with TextProvider"): - await transport.call_tool("some_tool", {}, provider) - - -@pytest.mark.asyncio -async def test_call_tool_file_not_found(transport: TextTransport): - """Test calling a tool when the file doesn't exist.""" - provider = TextProvider( - name="missing_file_provider", - file_path="/path/that/does/not/exist.json" - ) - - with pytest.raises(FileNotFoundError): - await transport.call_tool("some_tool", {}, provider) - - -@pytest.mark.asyncio -async def test_deregister_provider(transport: TextTransport, sample_utcp_manual): - """Test deregistering a text provider.""" - with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: - json.dump(sample_utcp_manual, f) - temp_file = f.name - - try: - provider = TextProvider( - name="test_provider", - file_path=temp_file - ) - - # Register and then deregister (should not raise any errors) - await transport.register_tool_provider(provider) - await transport.deregister_tool_provider(provider) - - finally: - Path(temp_file).unlink() - - -@pytest.mark.asyncio -async def test_close_transport(transport: TextTransport): - """Test closing the transport.""" - # Should not raise any errors - await transport.close() diff --git a/tests/client/transport_interfaces/test_udp_transport.py b/tests/client/transport_interfaces/test_udp_transport.py deleted file mode 100644 index 1bb3b0b..0000000 --- a/tests/client/transport_interfaces/test_udp_transport.py +++ /dev/null @@ -1,625 +0,0 @@ -import pytest -import pytest_asyncio -import json -import asyncio -import socket -from unittest.mock import MagicMock, patch, AsyncMock - -from utcp.client.transport_interfaces.udp_transport import UDPTransport -from utcp.shared.provider import UDPProvider -from utcp.shared.tool import Tool, ToolInputOutputSchema - - -class MockUDPServer: - """Mock UDP server for testing.""" - - def __init__(self, host='localhost', port=0): - self.host = host - self.port = port - self.sock = None - self.running = False - self.responses = {} # Map message -> response - self.call_count = 0 - self.listen_task = None - - async def start(self): - """Start the mock UDP server.""" - # Create socket and bind - self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) - # Keep it blocking since we're using run_in_executor - self.sock.bind((self.host, self.port)) - if self.port == 0: # Auto-assign port - self.port = self.sock.getsockname()[1] - - self.running = True - - # Start listening task - self.listen_task = asyncio.create_task(self._listen()) - - # Give the server a moment to start - await asyncio.sleep(0.1) - - async def stop(self): - """Stop the mock UDP server.""" - self.running = False - if self.listen_task: - self.listen_task.cancel() - try: - await self.listen_task - except asyncio.CancelledError: - pass - if self.sock: - self.sock.close() - - async def _listen(self): - """Listen for UDP messages and send responses.""" - # Use a blocking approach with short timeout for responsiveness - self.sock.settimeout(0.01) # Very short timeout - - while self.running: - try: - data, addr = self.sock.recvfrom(4096) - self.call_count += 1 - - try: - message = data.decode('utf-8') - except UnicodeDecodeError: - message = data.hex() # Fallback for binary data - - # Get response for this message - response = self.responses.get(message, '{"error": "unknown_message"}') - - # Convert response to bytes - if isinstance(response, str): - response_bytes = response.encode('utf-8') - elif isinstance(response, bytes): - response_bytes = response - elif isinstance(response, dict) or isinstance(response, list): - response_bytes = json.dumps(response).encode('utf-8') - else: - response_bytes = str(response).encode('utf-8') - - # Send response back immediately - self.sock.sendto(response_bytes, addr) - - except socket.timeout: - # Expected timeout, continue loop - await asyncio.sleep(0.001) # Brief async yield - continue - except asyncio.CancelledError: - break - except Exception as e: - if self.running: # Only log if we're still supposed to be running - import traceback - print(f"Mock UDP server error: {e}") - print(f"Traceback: {traceback.format_exc()}") - await asyncio.sleep(0.01) # Brief pause before retrying - - def set_response(self, message, response): - """Set a response for a specific message.""" - self.responses[message] = response - - -@pytest_asyncio.fixture -async def mock_udp_server(): - """Create a mock UDP server for testing.""" - server = MockUDPServer() - await server.start() - yield server - await server.stop() - - -@pytest.fixture -def logger(): - """Create a mock logger.""" - return MagicMock() - - -@pytest.fixture -def udp_transport(logger): - """Create a UDP transport instance.""" - return UDPTransport(logger=logger) - - -@pytest.fixture -def udp_provider(mock_udp_server): - """Create a basic UDP provider for testing.""" - return UDPProvider( - name="test_udp_provider", - host=mock_udp_server.host, - port=mock_udp_server.port, - number_of_response_datagrams=1, - request_data_format="json", - response_byte_format="utf-8", - timeout=5000 - ) - - -@pytest.fixture -def text_template_provider(mock_udp_server): - """Create a UDP provider with text template format.""" - return UDPProvider( - name="test_text_template_provider", - host=mock_udp_server.host, - port=mock_udp_server.port, - number_of_response_datagrams=1, - request_data_format="text", - request_data_template="COMMAND UTCP_ARG_action_UTCP_ARG UTCP_ARG_value_UTCP_ARG", - response_byte_format="utf-8", - timeout=5000 - ) - - -@pytest.fixture -def raw_bytes_provider(mock_udp_server): - """Create a UDP provider that returns raw bytes.""" - return UDPProvider( - name="test_raw_bytes_provider", - host=mock_udp_server.host, - port=mock_udp_server.port, - number_of_response_datagrams=1, - request_data_format="json", - response_byte_format=None, # Return raw bytes - timeout=5000 - ) - - -@pytest.fixture -def multi_datagram_provider(mock_udp_server): - """Create a UDP provider that expects multiple response datagrams.""" - return UDPProvider( - name="test_multi_datagram_provider", - host=mock_udp_server.host, - port=mock_udp_server.port, - number_of_response_datagrams=3, - request_data_format="json", - response_byte_format="utf-8", - timeout=5000 - ) - - -# Test register_tool_provider -@pytest.mark.asyncio -async def test_register_tool_provider(udp_transport, udp_provider, mock_udp_server, logger): - """Test registering a tool provider.""" - # Set up discovery response - discovery_response = { - "tools": [ - { - "name": "test_tool", - "description": "Test tool", - "inputs": { - "type": "object", - "properties": { - "param1": {"type": "string"} - } - }, - "outputs": { - "type": "object", - "properties": { - "result": {"type": "string"} - } - }, - "tags": [], - "tool_provider": { - "provider_type": "udp", - "name": "test_udp_provider", - "host": "localhost", - "port": udp_provider.port - } - } - ] - } - - mock_udp_server.set_response('{"type": "utcp"}', discovery_response) - print(f"Mock UDP server port: {mock_udp_server.port}") - print(f"UDP provider port: {udp_provider.port}") - - # Register the provider - tools = await udp_transport.register_tool_provider(udp_provider) - - # Verify tools were returned - assert len(tools) == 1 - assert tools[0].name == "test_tool" - assert tools[0].description == "Test tool" - - # Verify logger was called - logger.assert_called() - - -@pytest.mark.asyncio -async def test_register_tool_provider_empty_response(udp_transport, udp_provider, mock_udp_server): - """Test registering a tool provider with empty response.""" - # Set up empty discovery response - mock_udp_server.set_response('{"type": "utcp"}', {"tools": []}) - - # Register the provider - tools = await udp_transport.register_tool_provider(udp_provider) - - # Verify no tools were returned - assert len(tools) == 0 - - -@pytest.mark.asyncio -async def test_register_tool_provider_invalid_json(udp_transport, udp_provider, mock_udp_server): - """Test registering a tool provider with invalid JSON response.""" - # Set up invalid JSON response - mock_udp_server.set_response('{"type": "utcp"}', "invalid json") - - # Register the provider - tools = await udp_transport.register_tool_provider(udp_provider) - - # Verify no tools were returned due to JSON error - assert len(tools) == 0 - - -@pytest.mark.asyncio -async def test_register_tool_provider_invalid_provider_type(udp_transport): - """Test registering a non-UDP provider raises ValueError.""" - from utcp.shared.provider import HttpProvider - - http_provider = HttpProvider( - name="test_http_provider", - url="http://example.com" - ) - - with pytest.raises(ValueError, match="UDPTransport can only be used with UDPProvider"): - await udp_transport.register_tool_provider(http_provider) - - -# Test deregister_tool_provider -@pytest.mark.asyncio -async def test_deregister_tool_provider(udp_transport, udp_provider): - """Test deregistering a tool provider (should be a no-op).""" - # This should not raise any exceptions - await udp_transport.deregister_tool_provider(udp_provider) - - -@pytest.mark.asyncio -async def test_deregister_tool_provider_invalid_type(udp_transport): - """Test deregistering a non-UDP provider raises ValueError.""" - from utcp.shared.provider import HttpProvider - - http_provider = HttpProvider( - name="test_http_provider", - url="http://example.com" - ) - - with pytest.raises(ValueError, match="UDPTransport can only be used with UDPProvider"): - await udp_transport.deregister_tool_provider(http_provider) - - -# Test call_tool with JSON format -@pytest.mark.asyncio -async def test_call_tool_json_format(udp_transport, udp_provider, mock_udp_server): - """Test calling a tool with JSON format.""" - # Set up tool call response - arguments = {"param1": "value1", "param2": 42} - expected_message = json.dumps(arguments) - response = {"result": "success", "data": "processed"} - - mock_udp_server.set_response(expected_message, response) - - # Call the tool - result = await udp_transport.call_tool("test_tool", arguments, udp_provider) - - # Verify response - assert result == json.dumps(response) - assert mock_udp_server.call_count >= 1 - - -@pytest.mark.asyncio -async def test_call_tool_text_template_format(udp_transport, text_template_provider, mock_udp_server): - """Test calling a tool with text template format.""" - # Set up tool call response - arguments = {"action": "get", "value": "data123"} - expected_message = "COMMAND get data123" # Template substitution - response = "SUCCESS: data123 retrieved" - - mock_udp_server.set_response(expected_message, response) - - # Call the tool - result = await udp_transport.call_tool("test_tool", arguments, text_template_provider) - - # Verify response - assert result == response - assert mock_udp_server.call_count >= 1 - - -@pytest.mark.asyncio -async def test_call_tool_text_format_no_template(udp_transport, mock_udp_server): - """Test calling a tool with text format but no template.""" - provider = UDPProvider( - name="test_provider", - host=mock_udp_server.host, - port=mock_udp_server.port, - request_data_format="text", - request_data_template=None, # No template - response_byte_format="utf-8", - number_of_response_datagrams=1 # Expect 1 response - ) - - # Set up tool call response - arguments = {"param1": "value1", "param2": "value2"} - expected_message = "value1 value2" # Fallback format - response = "OK" - - mock_udp_server.set_response(expected_message, response) - - # Call the tool - result = await udp_transport.call_tool("test_tool", arguments, provider) - - # Verify response - assert result == response - - -@pytest.mark.asyncio -async def test_call_tool_raw_bytes_response(udp_transport, raw_bytes_provider, mock_udp_server): - """Test calling a tool that returns raw bytes.""" - # Set up tool call response with raw bytes - arguments = {"param1": "value1"} - expected_message = json.dumps(arguments) - raw_response = b"\x01\x02\x03\x04binary_data" - - mock_udp_server.set_response(expected_message, raw_response) - - # Call the tool - result = await udp_transport.call_tool("test_tool", arguments, raw_bytes_provider) - - # Verify response is raw bytes - assert isinstance(result, bytes) - assert result == raw_response - - -@pytest.mark.asyncio -async def test_call_tool_invalid_provider_type(udp_transport): - """Test calling a tool with non-UDP provider raises ValueError.""" - from utcp.shared.provider import HttpProvider - - http_provider = HttpProvider( - name="test_http_provider", - url="http://example.com" - ) - - with pytest.raises(ValueError, match="UDPTransport can only be used with UDPProvider"): - await udp_transport.call_tool("test_tool", {"param": "value"}, http_provider) - - -# Test multi-datagram support -@pytest.mark.asyncio -async def test_call_tool_multiple_datagrams(udp_transport, multi_datagram_provider, mock_udp_server): - """Test calling a tool that expects multiple response datagrams.""" - # This test is complex because we need to simulate multiple UDP responses - # For now, let's test that the transport handles the configuration correctly - - # Mock the _send_udp_message method to simulate multiple datagram responses - with patch.object(udp_transport, '_send_udp_message') as mock_send: - mock_send.return_value = "part1part2part3" # Concatenated response - - arguments = {"param1": "value1"} - result = await udp_transport.call_tool("test_tool", arguments, multi_datagram_provider) - - # Verify the method was called with correct parameters - mock_send.assert_called_once_with( - multi_datagram_provider.host, - multi_datagram_provider.port, - json.dumps(arguments), - multi_datagram_provider.timeout / 1000.0, - 3, # number_of_response_datagrams - "utf-8" # response_byte_format - ) - - assert result == "part1part2part3" - - -# Test _send_udp_message method directly -@pytest.mark.asyncio -async def test_send_udp_message_single_datagram(udp_transport, mock_udp_server): - """Test sending a UDP message and receiving a single response.""" - # Set up response - message = "test message" - response = "test response" - mock_udp_server.set_response(message, response) - - # Send message - result = await udp_transport._send_udp_message( - mock_udp_server.host, - mock_udp_server.port, - message, - timeout=5.0, - num_response_datagrams=1, - response_encoding="utf-8" - ) - - # Verify response - assert result == response - - -@pytest.mark.asyncio -async def test_send_udp_message_raw_bytes(udp_transport, mock_udp_server): - """Test sending a UDP message and receiving raw bytes.""" - # Set up binary response - message = "test message" - response = b"\x01\x02\x03binary" - mock_udp_server.set_response(message, response) - - # Send message with no encoding (raw bytes) - result = await udp_transport._send_udp_message( - mock_udp_server.host, - mock_udp_server.port, - message, - timeout=5.0, - num_response_datagrams=1, - response_encoding=None - ) - - # Verify response is bytes - assert isinstance(result, bytes) - assert result == response - - -@pytest.mark.asyncio -async def test_send_udp_message_timeout(): - """Test UDP message timeout handling.""" - udp_transport = UDPTransport() - - # Try to send to a non-existent server (should timeout) - with pytest.raises(Exception): # Should raise socket timeout or connection error - await udp_transport._send_udp_message( - "127.0.0.1", - 99999, # Non-existent port - "test message", - timeout=0.1, # Very short timeout - num_response_datagrams=1, - response_encoding="utf-8" - ) - - -# Test _format_tool_call_message method -def test_format_tool_call_message_json(udp_transport): - """Test formatting tool call message with JSON format.""" - provider = UDPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="json" - ) - - arguments = {"param1": "value1", "param2": 42} - result = udp_transport._format_tool_call_message(arguments, provider) - - # Should return JSON string - assert result == json.dumps(arguments) - - # Verify it's valid JSON - parsed = json.loads(result) - assert parsed == arguments - - -def test_format_tool_call_message_text_with_template(udp_transport): - """Test formatting tool call message with text template.""" - provider = UDPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="text", - request_data_template="ACTION UTCP_ARG_cmd_UTCP_ARG PARAM UTCP_ARG_value_UTCP_ARG" - ) - - arguments = {"cmd": "get", "value": "data123"} - result = udp_transport._format_tool_call_message(arguments, provider) - - # Should substitute placeholders - assert result == "ACTION get PARAM data123" - - -def test_format_tool_call_message_text_with_complex_values(udp_transport): - """Test formatting tool call message with complex values in template.""" - provider = UDPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="text", - request_data_template="DATA UTCP_ARG_obj_UTCP_ARG" - ) - - arguments = {"obj": {"nested": "value", "number": 123}} - result = udp_transport._format_tool_call_message(arguments, provider) - - # Should JSON-serialize complex values - assert result == 'DATA {"nested": "value", "number": 123}' - - -def test_format_tool_call_message_text_no_template(udp_transport): - """Test formatting tool call message with text format but no template.""" - provider = UDPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="text", - request_data_template=None - ) - - arguments = {"param1": "value1", "param2": "value2"} - result = udp_transport._format_tool_call_message(arguments, provider) - - # Should use fallback format (space-separated values) - assert result == "value1 value2" - - -def test_format_tool_call_message_default_to_json(udp_transport): - """Test formatting tool call message defaults to JSON for unknown format.""" - # Create a provider with valid format first - provider = UDPProvider( - name="test", - host="localhost", - port=1234, - request_data_format="json" - ) - - # Manually set an invalid format to test the fallback behavior - provider.request_data_format = "unknown" # Invalid format - - arguments = {"param1": "value1"} - result = udp_transport._format_tool_call_message(arguments, provider) - - # Should default to JSON - assert result == json.dumps(arguments) - - -# Test error handling and edge cases -@pytest.mark.asyncio -async def test_call_tool_server_error(udp_transport, udp_provider, mock_udp_server): - """Test handling server errors during tool calls.""" - # Don't set any response, so the server will return an error - arguments = {"param1": "value1"} - - # Call the tool - should get the default error response - result = await udp_transport.call_tool("test_tool", arguments, udp_provider) - - # Should receive the default error message - assert '{"error": "unknown_message"}' in result - - -@pytest.mark.asyncio -async def test_register_tool_provider_malformed_tool(udp_transport, udp_provider, mock_udp_server): - """Test registering provider with malformed tool definition.""" - # Set up discovery response with invalid tool - discovery_response = { - "tools": [ - { - "name": "test_tool", - # Missing required fields like inputs, outputs, tool_provider - } - ] - } - - mock_udp_server.set_response('{"type": "utcp"}', discovery_response) - - # Register the provider - should handle invalid tool gracefully - tools = await udp_transport.register_tool_provider(udp_provider) - - # Should return empty list due to invalid tool definition - assert len(tools) == 0 - - -# Test logging functionality -@pytest.mark.asyncio -async def test_logging_calls(udp_transport, udp_provider, mock_udp_server, logger): - """Test that logging functions are called appropriately.""" - # Set up discovery response - discovery_response = {"tools": []} - mock_udp_server.set_response('{"type": "utcp"}', discovery_response) - - # Register provider - await udp_transport.register_tool_provider(udp_provider) - - # Verify logger was called - logger.assert_called() - - # Call tool - mock_udp_server.set_response('{}', {"result": "test"}) - await udp_transport.call_tool("test_tool", {}, udp_provider) - - # Logger should have been called multiple times - assert logger.call_count > 1