diff --git a/docs/api/core/utcp/exceptions/utcp_serializer_validation_error.md b/docs/api/core/utcp/exceptions/utcp_serializer_validation_error.md index 4dcaaec..378f221 100644 --- a/docs/api/core/utcp/exceptions/utcp_serializer_validation_error.md +++ b/docs/api/core/utcp/exceptions/utcp_serializer_validation_error.md @@ -9,6 +9,20 @@ sidebar_label: utcp_serializer_validation_error ### class UtcpSerializerValidationError {#utcpserializervalidationerror} -*No class documentation available* +
+Documentation + +Exception raised when a serializer validation fails. + +Thrown by serializers when they cannot validate or convert data structures +due to invalid format, missing required fields, or type mismatches. +Contains the original validation error details for debugging. + + +**Usage** + +Typically caught when loading configuration files or processing +external data that doesn't conform to UTCP specifications. +
--- diff --git a/docs/api/core/utcp/implementations/post_processors/filter_dict_post_processor.md b/docs/api/core/utcp/implementations/post_processors/filter_dict_post_processor.md new file mode 100644 index 0000000..6193415 --- /dev/null +++ b/docs/api/core/utcp/implementations/post_processors/filter_dict_post_processor.md @@ -0,0 +1,49 @@ +--- +title: filter_dict_post_processor +sidebar_label: filter_dict_post_processor +--- + +# filter_dict_post_processor + +**File:** `core/src/utcp/implementations/post_processors/filter_dict_post_processor.py` + +### class FilterDictPostProcessor ([ToolPostProcessor](./../../interfaces/tool_post_processor.md#toolpostprocessor)) {#filterdictpostprocessor} + +
+Documentation + +Post-processor that filters dictionary keys from tool results. + +Provides flexible filtering capabilities to include or exclude specific keys +from dictionary results, with support for nested dictionaries and lists. +Can be configured to apply filtering only to specific tools or manuals. + + +**Attributes** + +- **`tool_post_processor_type`**: Always "filter_dict" for this processor. +- **`exclude_keys`**: List of keys to remove from dictionary results. +- **`only_include_keys`**: List of keys to keep in dictionary results (all others removed). +- **`exclude_tools`**: List of tool names to skip processing for. +- **`only_include_tools`**: List of tool names to process (all others skipped). +- **`exclude_manuals`**: List of manual names to skip processing for. +- **`only_include_manuals`**: List of manual names to process (all others skipped). +
+ +#### Fields: + +- tool_post_processor_type: Literal['filter_dict'] +- exclude_keys: Optional[List[str]] +- only_include_keys: Optional[List[str]] +- exclude_tools: Optional[List[str]] +- only_include_tools: Optional[List[str]] +- exclude_manuals: Optional[List[str]] +- only_include_manuals: Optional[List[str]] + +--- + +### class FilterDictPostProcessorConfigSerializer ([Serializer](./../../interfaces/serializer.md#serializer)[FilterDictPostProcessor]) {#filterdictpostprocessorconfigserializer} + +*No class documentation available* + +--- diff --git a/docs/api/core/utcp/implementations/post_processors/limit_strings_post_processor.md b/docs/api/core/utcp/implementations/post_processors/limit_strings_post_processor.md new file mode 100644 index 0000000..f524d48 --- /dev/null +++ b/docs/api/core/utcp/implementations/post_processors/limit_strings_post_processor.md @@ -0,0 +1,48 @@ +--- +title: limit_strings_post_processor +sidebar_label: limit_strings_post_processor +--- + +# limit_strings_post_processor + +**File:** `core/src/utcp/implementations/post_processors/limit_strings_post_processor.py` + +### class LimitStringsPostProcessor ([ToolPostProcessor](./../../interfaces/tool_post_processor.md#toolpostprocessor)) {#limitstringspostprocessor} + +
+Documentation + +Post-processor that limits the length of string values in tool results. + +Truncates string values to a specified maximum length to prevent +excessively large responses. Processes nested dictionaries and lists +recursively. Can be configured to apply limiting only to specific +tools or manuals. + + +**Attributes** + +- **`tool_post_processor_type`**: Always "limit_strings" for this processor. +- **`limit`**: Maximum length for string values (default: 10000 characters). +- **`exclude_tools`**: List of tool names to skip processing for. +- **`only_include_tools`**: List of tool names to process (all others skipped). +- **`exclude_manuals`**: List of manual names to skip processing for. +- **`only_include_manuals`**: List of manual names to process (all others skipped). +
+ +#### Fields: + +- tool_post_processor_type: Literal['limit_strings'] +- limit: int +- exclude_tools: Optional[List[str]] +- only_include_tools: Optional[List[str]] +- exclude_manuals: Optional[List[str]] +- only_include_manuals: Optional[List[str]] + +--- + +### class LimitStringsPostProcessorConfigSerializer ([Serializer](./../../interfaces/serializer.md#serializer)[LimitStringsPostProcessor]) {#limitstringspostprocessorconfigserializer} + +*No class documentation available* + +--- diff --git a/docs/api/core/utcp/implementations/tag_search.md b/docs/api/core/utcp/implementations/tag_search.md index af032ab..558045e 100644 --- a/docs/api/core/utcp/implementations/tag_search.md +++ b/docs/api/core/utcp/implementations/tag_search.md @@ -9,7 +9,38 @@ sidebar_label: tag_search ### class TagAndDescriptionWordMatchStrategy ([ToolSearchStrategy](./../interfaces/tool_search_strategy.md#toolsearchstrategy)) {#taganddescriptionwordmatchstrategy} -*No class documentation available* +
+Documentation + +Tag and description word match strategy. + + +**Implements A Weighted Scoring System That Matches Tools Based On** + +1. Tag matches (higher weight) +2. Description word matches (lower weight) + +The strategy normalizes queries to lowercase, extracts words using regex, +and calculates relevance scores for each tool. Results are sorted by +score in descending order. + + + +**Attributes** + +- **`tool_search_strategy_type`**: Always "tag_and_description_word_match". +- **`description_weight`**: Weight multiplier for description word matches (default: 1.0). +- **`tag_weight`**: Weight multiplier for tag matches (default: 3.0). + + + +**Scoring Algorithm** + +- Each matching tag contributes tag_weight points +- Each matching description word contributes description_weight points +- Tools with higher scores are ranked first +- Tools with zero score are included in results (ranked last) +
#### Fields: diff --git a/docs/api/core/utcp/plugins/plugin_loader.md b/docs/api/core/utcp/plugins/plugin_loader.md index 8e8e9c1..3ac6524 100644 --- a/docs/api/core/utcp/plugins/plugin_loader.md +++ b/docs/api/core/utcp/plugins/plugin_loader.md @@ -7,6 +7,12 @@ sidebar_label: plugin_loader **File:** `core/src/utcp/plugins/plugin_loader.py` +### Function _load_plugins() {#_load_plugins} + +*No function documentation available* + +--- + ### Function ensure_plugins_initialized() {#ensure_plugins_initialized} *No function documentation available* diff --git a/docs/api/index.md b/docs/api/index.md index 51585f8..50ef8a5 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -11,8 +11,8 @@ This specification is organized by module of the reference python implementation **Note:** The modules don't have to be implemented in the same way as in the reference implementation, but all of the functionality here needs to be provided. -**Total documented items:** 189 -**Modules documented:** 39 +**Total documented items:** 195 +**Modules documented:** 41 ## Core Modules @@ -93,6 +93,16 @@ Core UTCP framework components that define the fundamental interfaces and implem - **Contains:** 2 classes, 12 methods +### [utcp.implementations.post_processors.filter_dict_post_processor](./core\utcp\implementations\post_processors\filter_dict_post_processor.md) + +- **Contains:** 2 classes + + +### [utcp.implementations.post_processors.limit_strings_post_processor](./core\utcp\implementations\post_processors\limit_strings_post_processor.md) + +- **Contains:** 2 classes + + ### [utcp.implementations.tag_search](./core\utcp\implementations\tag_search.md) - **Contains:** 2 classes, 3 methods @@ -140,7 +150,7 @@ Core UTCP framework components that define the fundamental interfaces and implem ### [utcp.plugins.plugin_loader](./core\utcp\plugins\plugin_loader.md) -- **Contains:** 1 functions +- **Contains:** 2 functions ### [utcp.utcp_client](./core\utcp\utcp_client.md) @@ -154,7 +164,7 @@ Plugin implementations that extend UTCP with specific transport protocols and ca ### [communication_protocols.cli.src.utcp_cli.cli_call_template](./plugins\communication_protocols\cli\src\utcp_cli\cli_call_template.md) -- **Contains:** 2 classes, 2 methods +- **Contains:** 3 classes, 2 methods ### [communication_protocols.cli.src.utcp_cli.cli_communication_protocol](./plugins\communication_protocols\cli\src\utcp_cli\cli_communication_protocol.md) diff --git a/docs/api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md b/docs/api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md index abfe4aa..54c7dca 100644 --- a/docs/api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md +++ b/docs/api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md @@ -7,30 +7,173 @@ sidebar_label: cli_call_template **File:** `plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py` +### class CommandStep {#commandstep} + +
+Documentation + +Configuration for a single command step in a CLI execution flow. + + +**Attributes** + +- **`command`**: The command string to execute. Can contain UTCP_ARG_argname_UTCP_END + placeholders that will be replaced with values from tool_args. Can also + reference previous command outputs using $CMD_0_OUTPUT, $CMD_1_OUTPUT, etc. +- **`append_to_final_output`**: Whether this command's output should be included + in the final result. If not specified, defaults to False for all + commands except the last one. + + + +**Basic Command Step** + +```json + { + "command": "git status", + "append_to_final_output": true + } +``` + + + +**Command With Argument Placeholders And Output Reference** + +```json + { + "command": "echo "Cloning to: UTCP_ARG_target_dir_UTCP_END, previous status: $CMD_0_OUTPUT"", + "append_to_final_output": true + } +``` +
+ +#### Fields: + +- command: str +- append_to_final_output: Optional[bool] + +--- + ### class CliCallTemplate ([CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) {#clicalltemplate}
Documentation -Call template configuration for Command Line Interface tools. +Call template configuration for Command Line Interface (CLI) tools. + +This class defines the configuration for executing command-line tools and +programs as UTCP tool providers. Commands are executed in a single subprocess +to maintain state (like directory changes) between commands. + + + + +**You Can Reference The Output Of Previous Commands Using Variables** + + +- **`Example`**: `echo "Previous result: $CMD_0_OUTPUT"` -Enables execution of command-line tools and programs as UTCP providers. -Supports environment variable injection and custom working directories. **Attributes** -- **`call_template_type`**: Always "cli" for CLI providers. -- **`command_name`**: The name or path of the command to execute. -- **`env_vars`**: Optional environment variables to set during command execution. -- **`working_dir`**: Optional custom working directory for command execution. -- **`auth`**: Always None - CLI providers don't support authentication. +- **`call_template_type`**: The type of the call template. Must be "cli". +- **`commands`**: A list of CommandStep objects defining the commands to execute + in order. Each command can contain UTCP_ARG_argname_UTCP_END placeholders + that will be replaced with values from tool_args during execution. +- **`env_vars`**: A dictionary of environment variables to set for the command's + execution context. Values can be static strings or placeholders for + variables from the UTCP client's variable substitutor. +- **`working_dir`**: The working directory from which to run the commands. If not + provided, it defaults to the current process's working directory. +- **`auth`**: Authentication details. Not applicable to the CLI protocol, so it + is always None. + + + +**Cross-Platform Directory Operations** + +```json + { + "name": "cross_platform_dir_tool", + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_target_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "ls -la", + "append_to_final_output": true + } + ] + } +``` + + + +**Referencing Previous Command Output** + +```json + { + "name": "reference_previous_output_tool", + "call_template_type": "cli", + "commands": [ + { + "command": "git status --porcelain", + "append_to_final_output": false + }, + { + "command": "echo "Found changes: $CMD_0_OUTPUT"", + "append_to_final_output": true + } + ] + } +``` + + + +**Command With Environment Variables And Placeholders** + +```json + { + "name": "python_multi_step_tool", + "call_template_type": "cli", + "commands": [ + { + "command": "python setup.py install", + "append_to_final_output": false + }, + { + "command": "python script.py --input UTCP_ARG_input_file_UTCP_END --result "$CMD_0_OUTPUT"" + } + ], + "env_vars": { + "PYTHONPATH": "/custom/path", + "API_KEY": "${API_KEY_VAR}" + } + } +``` + + + +**Security Considerations** + +- Commands are executed in a subprocess. Ensure that the commands +specified are from a trusted source. +- Avoid passing unsanitized user input directly into the command string. +Use tool argument validation where possible. +- All placeholders are replaced with string values from tool_args. +- Commands should use the appropriate syntax for the target platform +(PowerShell on Windows, Bash on Unix). +- Previous command outputs are available as variables but should be +used carefully to avoid command injection.
#### Fields: - call_template_type: Literal['cli'] -- command_name: str +- commands: List[CommandStep] - env_vars: Optional[Dict[str, str]] - working_dir: Optional[str] - auth: None @@ -46,13 +189,42 @@ Supports environment variable injection and custom working directories.
to_dict(self, obj: CliCallTemplate) -> dict -*No method documentation available* +Converts a `CliCallTemplate` instance to its dictionary representation. + + +**Args** + +- **`obj`**: The `CliCallTemplate` instance to serialize. + + + +**Returns** + +A dictionary representing the `CliCallTemplate`.
validate_dict(self, obj: dict) -> CliCallTemplate -*No method documentation available* +Validates a dictionary and constructs a `CliCallTemplate` instance. + + +**Args** + +- **`obj`**: The dictionary to validate and deserialize. + + + +**Returns** + +A `CliCallTemplate` instance. + + + +**Raises** + +- **`[UtcpSerializerValidationError](./../../../../../core/utcp/exceptions/utcp_serializer_validation_error.md#utcpserializervalidationerror)`**: If the dictionary is not a valid + representation of a `CliCallTemplate`.
--- diff --git a/docs/api/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.md b/docs/api/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.md index ee2e071..5ed1941 100644 --- a/docs/api/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.md +++ b/docs/api/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.md @@ -9,87 +9,98 @@ sidebar_label: cli_communication_protocol ### class CliCommunicationProtocol ([CommunicationProtocol](./../../../../../core/utcp/interfaces/communication_protocol.md#communicationprotocol)) {#clicommunicationprotocol} -
-Documentation - -Transport implementation for CLI-based tool providers. +*No class documentation available* -Handles communication with command-line tools by executing processes -and managing their input/output. Supports both tool discovery and -execution phases with comprehensive error handling and timeout management. +#### Methods: +
+async register_manual(self, caller, manual_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> [RegisterManualResult](./../../../../../core/utcp/data/register_manual_response.md#registermanualresult) -**Features** +Registers a CLI-based manual and discovers its tools. -- Asynchronous subprocess execution with proper cleanup -- [Tool](./../../../../../core/utcp/data/tool.md#tool) discovery through startup commands returning UTCP manuals -- Flexible argument formatting for various CLI conventions -- Environment variable injection for authentication -- JSON output parsing with graceful fallback to text -- Cross-platform command parsing and execution -- Configurable working directories and timeouts -- Process lifecycle management with proper termination +This method executes the command specified in the `[CliCallTemplate](./cli_call_template.md#clicalltemplate)`'s +`command_name` field. It then attempts to parse the command's output +(stdout) as a UTCP manual in JSON format. +**Args** -**Architecture** +- **`caller`**: The UTCP client instance that is calling this method. +- **`manual_call_template`**: The `[CliCallTemplate](./cli_call_template.md#clicalltemplate)` containing the details for + tool discovery, such as the command to run. -CLI tools are discovered by executing the provider's command_name -and parsing the output for UTCP manual JSON. [Tool](./../../../../../core/utcp/data/tool.md#tool) calls execute -the same command with formatted arguments and return processed output. +**Returns** -**Attributes** +A `[RegisterManualResult](./../../../../../core/utcp/data/register_manual_response.md#registermanualresult)` object indicating whether the registration +was successful and containing the discovered tools. -- **`_log`**: Logger function for debugging and error reporting. -
-#### Methods: -
-async register_manual(self, caller, manual_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> [RegisterManualResult](./../../../../../core/utcp/data/register_manual_response.md#registermanualresult) +**Raises** -*No method documentation available* +- **`ValueError`**: If the `manual_call_template` is not an instance of + `[CliCallTemplate](./cli_call_template.md#clicalltemplate)` or if `command_name` is not set.
async deregister_manual(self, caller, manual_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> None -*No method documentation available* +Deregisters a CLI manual. + +For the CLI protocol, this is a no-op as there are no persistent +connections to terminate. + + +**Args** + +- **`caller`**: The UTCP client instance that is calling this method. +- **`manual_call_template`**: The call template of the manual to deregister.
async call_tool(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> Any -Call a CLI tool. +Calls a CLI tool by executing its command. -Executes the command specified by provider.command_name with the provided arguments. +This method constructs and executes the command specified in the +`[CliCallTemplate](./cli_call_template.md#clicalltemplate)`. It formats the provided `tool_args` as command-line +arguments and runs the command in a subprocess. **Args** -- **`caller`**: The UTCP client that is calling this method. -- **`tool_name`**: Name of the tool to call -- **`tool_args`**: Arguments for the tool call -- **`tool_call_template`**: The [CliCallTemplate](./cli_call_template.md#clicalltemplate) for the tool +- **`caller`**: The UTCP client instance that is calling this method. +- **`tool_name`**: The name of the tool to call. +- **`tool_args`**: A dictionary of arguments for the tool call. +- **`tool_call_template`**: The `[CliCallTemplate](./cli_call_template.md#clicalltemplate)` for the tool. -**The Output From The Command Execution Based On Exit Code** +**Returns** +The result of the command execution. If the command exits with a code +of 0, it returns the content of stdout. If the exit code is non-zero, +it returns the content of stderr. **Raises** -- **`ValueError`**: If provider is not a CliProvider or command_name is not set +- **`ValueError`**: If `tool_call_template` is not an instance of + `[CliCallTemplate](./cli_call_template.md#clicalltemplate)` or if `command_name` is not set.
async call_tool_streaming(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> AsyncGenerator[Any, None] -*No method documentation available* +Streaming calls are not supported for the CLI protocol. + + +**Raises** + +- **`NotImplementedError`**: Always, as this functionality is not supported.
--- diff --git a/docs/api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md b/docs/api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md index 2420d59..98dda8c 100644 --- a/docs/api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md +++ b/docs/api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md @@ -20,6 +20,81 @@ parameters using \{parameter_name\} syntax. All tool arguments not mapped to URL body, headers or query pattern parameters are passed as query parameters using '?arg_name=\{arg_value\}'. +**Basic Http Get Request** + +```json + { + "name": "my_rest_api", + "call_template_type": "http", + "url": "https://api.example.com/users/{user_id}", + "http_method": "GET" + } +``` + + + +**Post With Authentication** + +```json + { + "name": "secure_api", + "call_template_type": "http", + "url": "https://api.example.com/users", + "http_method": "POST", + "content_type": "application/json", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_KEY}", + "var_name": "Authorization", + "location": "header" + }, + "headers": { + "X-Custom-Header": "value" + }, + "body_field": "body", + "header_fields": ["user_id"] + } +``` + + + +**Oauth2 Authentication** + +```json + { + "name": "oauth_api", + "call_template_type": "http", + "url": "https://api.example.com/data", + "http_method": "GET", + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token" + } + } +``` + + + +**Basic Authentication** + +```json + { + "name": "basic_auth_api", + "call_template_type": "http", + "url": "https://api.example.com/secure", + "http_method": "GET", + "auth": { + "auth_type": "basic", + "username": "${USERNAME}", + "password": "${PASSWORD}" + } + } +``` + + + **Attributes** - **`call_template_type`**: Always "http" for HTTP providers. diff --git a/docs/api/plugins/communication_protocols/http/src/utcp_http/openapi_converter.md b/docs/api/plugins/communication_protocols/http/src/utcp_http/openapi_converter.md index 0a68759..dd41955 100644 --- a/docs/api/plugins/communication_protocols/http/src/utcp_http/openapi_converter.md +++ b/docs/api/plugins/communication_protocols/http/src/utcp_http/openapi_converter.md @@ -22,14 +22,50 @@ a UTCP tool with appropriate input/output schemas. **Features** -- Complete OpenAPI specification parsing -- Recursive JSON reference ($ref) resolution -- Authentication scheme conversion (API key, Basic, OAuth2) -- Input parameter and request body handling -- Response schema extraction -- URL template and path parameter support -- Provider name normalization -- Placeholder variable generation for configuration +- Complete OpenAPI specification parsing. +- Recursive JSON reference ($ref) resolution. +- Authentication scheme conversion (API key, Basic, OAuth2). +- Input parameter and request body handling. +- Response schema extraction. +- URL template and path parameter support. +- Call template name normalization. +- Placeholder variable generation for configuration. + + + +**Basic Openapi Conversion** + +```python + from utcp_http.openapi_converter import OpenApiConverter + + # Assuming you have a method to fetch and parse the spec + openapi_spec = fetch_and_parse_spec("https://api.example.com/openapi.json") + + converter = OpenApiConverter(openapi_spec) + manual = converter.convert() + + # Use the generated manual with a UTCP client + # client = await UtcpClient.create() + # await client.register_manual(manual) +``` + + + +**Converting Local Openapi File** + +```python + import yaml + + converter = OpenApiConverter() + + +**With Open("Api_Spec.Yaml", "R") As F** + +spec_content = yaml.safe_load(f) + +converter = OpenApiConverter(spec_content) +manual = converter.convert() +``` @@ -54,7 +90,16 @@ operation, and creating corresponding UTCP tools with HTTP call_templates.
convert(self) -> [UtcpManual](./../../../../../core/utcp/data/utcp_manual.md#utcpmanual) -*No method documentation available* +Converts the loaded OpenAPI specification into a [UtcpManual](./../../../../../core/utcp/data/utcp_manual.md#utcpmanual). + +This is the main entry point for the conversion process. It iterates through +the paths and operations in the specification, creating a UTCP tool for each +one. + + +**Returns** + +A [UtcpManual](./../../../../../core/utcp/data/utcp_manual.md#utcpmanual) object containing all the tools generated from the spec.
--- diff --git a/docs/api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md b/docs/api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md index 2b36fbb..8f5cc77 100644 --- a/docs/api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md +++ b/docs/api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md @@ -44,12 +44,109 @@ interfaces. Supports both stdio (local process) and HTTP (remote) transport methods. +**Basic Mcp Server With Stdio Transport** + +```json + { + "name": "mcp_server", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "filesystem": { + "command": "node", + "args": ["mcp-server.js"], + "env": {"NODE_ENV": "production"} + } + } + } + } +``` + + + +**Mcp Server With Working Directory** + +```json + { + "name": "mcp_tools", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "tools": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/app/mcp", + "env": { + "PYTHONPATH": "/app", + "LOG_LEVEL": "INFO" + } + } + } + } + } +``` + + + +**Mcp Server With Oauth2 Authentication** + +```json + { + "name": "secure_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "secure_server": { + "transport": "http", + "url": "https://mcp.example.com" + } + } + }, + "auth": { + "auth_type": "oauth2", + "token_url": "https://auth.example.com/token", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "scope": "read:tools" + } + } +``` + + + +**During Migration (Utcp With Mcp)** + +```python + # UTCP Client with MCP plugin + client = await UtcpClient.create() + result = await client.call_tool("filesystem.read_file", { + "path": "/data/file.txt" + }) +``` + + + +**After Migration (Pure Utcp)** + +```python + # UTCP Client with native protocol + client = await UtcpClient.create() + result = await client.call_tool("filesystem.read_file", { + "path": "/data/file.txt" + }) +``` + + + **Attributes** - **`call_template_type`**: Always "mcp" for MCP providers. - **`config`**: Configuration object containing MCP server definitions. This follows the same format as the official MCP server configuration. - **`auth`**: Optional OAuth2 authentication for HTTP-based MCP servers. +- **`register_resources_as_tools`**: Whether to register MCP resources as callable tools. + When True, server resources are exposed as tools that can be called. + Default is False.
#### Fields: @@ -57,6 +154,7 @@ transport methods. - call_template_type: Literal['mcp'] - config: McpConfig - auth: Optional[[OAuth2Auth](./../../../../../core/utcp/data/auth_implementations/oauth2_auth.md#oauth2auth)] +- register_resources_as_tools: bool --- diff --git a/docs/for-tool-providers.md b/docs/for-tool-providers.md new file mode 100644 index 0000000..87ff739 --- /dev/null +++ b/docs/for-tool-providers.md @@ -0,0 +1,580 @@ +--- +id: for-tool-providers +title: For Tool Providers +sidebar_position: 2 +--- + +# For Tool Providers + +:::info Language Note +This guide covers how to expose your existing APIs and services as UTCP tools. UTCP implementations are available in multiple languages - check the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) for Python, TypeScript, Go, and other language implementations. +::: + +This guide helps you expose your tools through UTCP so they can be discovered and used by AI agents and other applications. + +## Overview + +As a tool provider, you'll create a **UTCP Manual** - a standardized description of your tools that tells clients how to call them directly using their native protocols. This eliminates the need for wrapper servers and allows direct communication. + +## Quick Start + +### 1. Create a Simple Manual + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [ + { + "name": "get_user", + "description": "Retrieve user information by ID", + "inputs": { + "type": "object", + "properties": { + "user_id": {"type": "string", "description": "User identifier"} + }, + "required": ["user_id"] + }, + "outputs": { + "type": "object", + "properties": { + "id": {"type": "string"}, + "name": {"type": "string"}, + "email": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users/${user_id}", + "http_method": "GET" + } + } + ] +} +``` + +### 2. Expose via Discovery Endpoint + +Create an HTTP endpoint that returns your UTCP manual: + +**Endpoint**: `GET /utcp` +**Response**: +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [ + // ... your tools here + ] +} +``` + +Your existing API endpoints remain unchanged. For example: +- `GET /users/{user_id}` - Returns user data +- `POST /orders` - Creates new orders +- etc. + +## Manual Structure + +The UTCP manual follows a standardized structure that defines your tools and how to call them. For complete field specifications, data types, and validation rules, see the [UTCP Manual API Reference](./api/core/utcp/data/utcp_manual.md). + +### Key Components + +- **Manual metadata**: Version information and API details +- **Tool definitions**: Description of available tools and their capabilities +- **Call templates**: Instructions for invoking each tool +- **Authentication**: Security configuration for tool access +- **Variables**: Dynamic values for tool parameters + +### Tool Definition + +Tools are defined in your UTCP manual with their input parameters, call instructions, and optional metadata. For complete field specifications, see the [Tool API Reference](./api/core/utcp/data/tool.md). + +## Communication Protocol Plugins + +### HTTP Tools + +Most common for REST APIs: + +```json +{ + "name": "create_user", + "description": "Create a new user account", + "inputs": { + "type": "object", + "properties": { + "user_fields": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "email": {"type": "string", "format": "email"} + }, + "required": ["name", "email"] + } + }, + "required": ["user_fields"] + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + }, + "body_field": "user_fields" + } +} +``` + +### CLI Tools + +For command-line applications: + +```json +{ + "name": "git_analysis", + "description": "Analyze git repository status and commit history", + "inputs": { + "type": "object", + "properties": { + "repo_path": {"type": "string", "description": "Path to git repository"} + }, + "required": ["repo_path"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_repo_path_UTCP_END", + "append_to_final_output": false + }, + { + "command": "git status --porcelain", + "append_to_final_output": false + }, + { + "command": "echo \"Status: $CMD_1_OUTPUT, Files changed: $(echo \"$CMD_1_OUTPUT\" | wc -l)\"", + "append_to_final_output": true + } + ] + } +} +``` + +## Authentication + +### API Key Authentication + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +### Bearer Token + +To use a Bearer Token, you can use the `api_key` authentication type. + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${ACCESS_TOKEN}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +### OAuth2 + +```json +{ + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "read:users write:users" + } +} +``` + +### Per-Tool Authentication + +```json +{ + "name": "admin_action", + "description": "Perform admin action", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/admin/action", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "${ADMIN_TOKEN}", + "var_name": "Authorization", + "location": "header" + } + } +} +``` + +## Variable Substitution + +Use `${VARIABLE_NAME}` syntax for dynamic values: + +### From Tool Arguments + +```json +{ + "url": "https://api.example.com/users/${user_id}", + "body": { + "name": "${name}", + "email": "${email}" + } +} +``` + +### From Environment Variables + +```json +{ + "headers": { + "Authorization": "Bearer ${API_TOKEN}", + "X-Client-ID": "${CLIENT_ID}" + } +} +``` + +### Default Values + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "variables": { + "base_url": "https://api.example.com", + "timeout": 30 + }, + "tools": [ + { + "name": "get_data", + "tool_call_template": { + "call_template_type": "http", + "url": "${base_url}/data", + "timeout": "${timeout}" + } + } + ] +} +``` + +## Implementation Examples + +### REST API Implementation + +For a typical REST API, you'll need to: + +1. **Keep your existing endpoints unchanged** +2. **Add a UTCP discovery endpoint** at `/utcp` +3. **Map your API operations to UTCP tools** + +Example API structure: +``` +GET /users/{user_id} # Your existing endpoint +POST /users # Your existing endpoint +GET /utcp # New UTCP discovery endpoint +``` + +The UTCP manual describes how to call your existing endpoints: + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "info": { + "title": "User Management API", + "version": "1.0.0", + "description": "API for managing user accounts" + }, + "tools": [ + { + "name": "get_user", + "description": "Retrieve user information by ID", + "inputs": { + "type": "object", + "properties": { + "user_id": {"type": "string"} + }, + "required": ["user_id"] + }, + "outputs": { + "type": "object", + "properties": { + "id": {"type": "string"}, + "name": {"type": "string"}, + "email": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users/${user_id}", + "http_method": "GET", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + } + } + }, + { + "name": "create_user", + "description": "Create a new user account", + "inputs": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "email": {"type": "string"} + }, + "required": ["name", "email"] + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + }, + "body_field": "user_data" + } + } + ] +} +``` + +## OpenAPI Integration + +If you already have an OpenAPI/Swagger specification, you can automatically convert it to a UTCP manual: + +### Automatic Conversion + +Many UTCP implementations provide OpenAPI converters that can: + +1. **Parse OpenAPI specifications** from URLs or files +2. **Convert paths to UTCP tools** automatically +3. **Map authentication schemes** to UTCP auth types +4. **Generate proper input/output schemas** + +### Conversion Configuration + +You can customize the conversion process: + +```json +{ + "source": "https://api.example.com/openapi.json", + "base_url": "https://api.example.com", + "include_operations": ["get", "post"], + "exclude_paths": ["/internal/*"], + "auth_template": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +### Manual Review + +After automatic conversion: +1. **Review generated tools** for accuracy +2. **Add missing descriptions** and examples +3. **Validate input/output schemas** +4. **Test with UTCP clients** +5. **Customize authentication** as needed + +## Best Practices + +### Manual Design + +1. **Clear Descriptions**: Write clear, concise tool descriptions +2. **Comprehensive Schemas**: Use detailed JSON schemas for inputs/outputs +3. **Consistent Naming**: Use consistent naming conventions +4. **Version Management**: Use semantic versioning for your manual +5. **Documentation**: Include examples and usage notes + +### Security + +1. **Authentication**: Always implement proper authentication +2. **Input Validation**: Validate all inputs on your API side +3. **Rate Limiting**: Implement rate limiting to prevent abuse +4. **HTTPS Only**: Use HTTPS for all production endpoints +5. **Credential Management**: Never hardcode credentials in manuals + +### Performance + +1. **Efficient Endpoints**: Design efficient API endpoints +2. **Caching**: Implement appropriate caching strategies +3. **Pagination**: Use pagination for large result sets +4. **Timeouts**: Set reasonable timeout values +5. **Monitoring**: Monitor API performance and usage + +### Maintenance + +1. **Versioning**: Version your manual and API together +2. **Backward Compatibility**: Maintain backward compatibility when possible +3. **Deprecation**: Provide clear deprecation notices +4. **Testing**: Test your manual with UTCP clients +5. **Documentation**: Keep documentation up to date + +## Testing Your Manual + +### Manual Validation + +Validate your UTCP manual structure: + +1. **JSON Schema Validation**: Ensure your manual follows the UTCP schema +2. **Tool Definition Validation**: Check that all tools have required fields +3. **Call Template Validation**: Verify call templates are properly formatted +4. **Authentication Validation**: Test authentication configurations + +### Integration Testing + +Test your manual with UTCP clients: + +1. **Tool Discovery**: Verify clients can discover your tools +2. **Tool Execution**: Test actual tool calls with various inputs +3. **Error Handling**: Test error scenarios and responses +4. **Authentication**: Verify authentication works correctly +5. **Performance**: Test response times and reliability + +### Testing Checklist + +- [ ] Manual validates against UTCP schema +- [ ] All tools have unique names +- [ ] All required fields are present +- [ ] Call templates are correctly formatted +- [ ] Authentication works as expected +- [ ] Tools return expected outputs +- [ ] Error responses are properly formatted +- [ ] Performance meets requirements + +## Migration Strategies + +### From OpenAPI + +1. **Automatic Conversion**: Use OpenAPI converter for initial conversion +2. **Manual Refinement**: Refine converted manual for better descriptions +3. **Authentication Setup**: Configure authentication properly +4. **Testing**: Test converted manual thoroughly + +### From MCP + +1. **Wrapper Approach**: Use UTCP-MCP plugin initially +2. **Gradual Migration**: Migrate tools one by one to native protocols +3. **Direct Implementation**: Implement tools using native UTCP protocols +4. **Deprecation**: Remove MCP dependency once migration is complete + +## Common Patterns + +### CRUD Operations + +```json +{ + "tools": [ + { + "name": "create_resource", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/resources", + "http_method": "POST" + } + }, + { + "name": "get_resource", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/resources/${id}", + "http_method": "GET" + } + }, + { + "name": "update_resource", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/resources/${id}", + "http_method": "PUT" + } + }, + { + "name": "delete_resource", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/resources/${id}", + "http_method": "DELETE" + } + } + ] +} +``` + +### Batch Operations + +```json +{ + "name": "batch_process", + "description": "Process multiple items in batch", + "inputs": { + "type": "object", + "properties": { + "items": { + "type": "array", + "items": {"type": "object"} + } + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/batch", + "http_method": "POST", + "body": { + "items": "${items}" + } + } +} +``` + +## Next Steps + +1. **Design Your Manual**: Plan your tool structure and descriptions +2. **Choose Protocols**: Select appropriate communication protocols +3. **Implement Discovery**: Add the `/utcp` endpoint to your API +4. **Test Integration**: Test with UTCP clients +5. **Monitor Usage**: Monitor how your tools are being used +6. **Iterate**: Improve based on usage patterns and feedback + +For more information, see: +- [Communication Protocol Plugins](./protocols/index.md) +- [Implementation Guide](./implementation.md) +- [Security Considerations](./security.md) + +## Language-Specific Implementation + +For detailed implementation examples and code samples in your programming language: + +- **Multi-language**: [UTCP Implementation Examples](https://github.com/universal-tool-calling-protocol) - Examples across Python, TypeScript, Go, and other languages +- **TypeScript**: [TypeScript UTCP Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/tree/main/docs) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) diff --git a/docs/implementation.md b/docs/implementation.md new file mode 100644 index 0000000..c1aee3a --- /dev/null +++ b/docs/implementation.md @@ -0,0 +1,396 @@ +--- +id: implementation +title: Implementation Guide +sidebar_position: 4 +--- + +# UTCP Implementation Guide + +This guide covers the core concepts and patterns for implementing UTCP in any programming language, whether you're building tool providers or tool consumers. + +## Quick Start + +### 1. Install UTCP Library + +Choose the UTCP implementation for your programming language: + +- **Python**: `pip install utcp utcp-http utcp-cli` +- **Node.js**: `npm install @utcp/core @utcp/http @utcp/cli` +- **Other languages**: Check the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) + +### 2. Create Your First Tool Provider + +Create an HTTP endpoint that serves a UTCP manual: + +**Endpoint**: `GET /utcp` +**Response**: +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "info": { + "title": "Weather API", + "version": "1.0.0" + }, + "tools": [ + { + "name": "get_weather", + "description": "Get current weather for a location", + "inputs": { + "type": "object", + "properties": { + "location": {"type": "string"} + }, + "required": ["location"] + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.weather.com/current", + "http_method": "GET", + "auth": { + "auth_type": "api_key", + "api_key": "${WEATHER_API_KEY}", + "var_name": "appid", + "location": "query" + } + } + } + ] +} +``` + +### 3. Create Your First Client + +Configure a UTCP client to discover and call tools: + +**Configuration**: +```json +{ + "manual_call_templates": [ + { + "name": "weather_service", + "call_template_type": "http", + "url": "https://api.weather.com/utcp", + "http_method": "GET" + } + ], + "variables": { + "WEATHER_API_KEY": "your-api-key" + } +} +``` + +**Usage**: +1. Initialize UTCP client with configuration +2. Discover tools from the weather service +3. Call the `get_weather` tool with location parameter + +## Core Concepts + +### UTCP Manual + +A UTCP manual is a JSON document that describes available tools and how to call them: + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "info": { + "title": "API Name", + "version": "1.0.0", + "description": "API description" + }, + "tools": [ + { + "name": "tool_name", + "description": "Tool description", + "inputs": { + "type": "object", + "properties": { + "param": {"type": "string"} + } + }, + "outputs": { + "type": "object", + "properties": { + "result": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "POST" + } + } + ] +} +``` + +### Call Templates + +Call templates define how to invoke tools using specific protocols: + +#### HTTP Call Template +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + }, + "body_field": "body" +} + +Tool arguments not used in the URL path or headers will be sent as query parameters for GET requests, or in the request body for POST/PUT/PATCH requests. The `body_field` specifies which tool argument contains the data for the request body. +} +``` + +#### CLI Call Template +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "cd /app", + "append_to_final_output": false + }, + { + "command": "python script.py UTCP_ARG_input_UTCP_END", + "append_to_final_output": true + } + ], + "env_vars": { + "PYTHONPATH": "/app/lib" + }, + "working_dir": "/app" +} +``` + +### Variable Substitution + +Variables in call templates are replaced with actual values: + +- **Tool arguments**: `${argument_name}` +- **Environment variables**: `${ENV_VAR}` +- **Configuration variables**: `${config.variable}` + +## Tool Provider Implementation + +### Manual Structure + +Create a well-structured UTCP manual: + +1. **Info Section**: Describe your API +2. **Tools Array**: Define each available tool +3. **Input Schemas**: Specify required parameters +4. **Output Schemas**: Document return values +5. **Call Templates**: Define how to invoke each tool + +### Discovery Endpoint + +Expose your manual via HTTP: + +``` +GET /utcp +Content-Type: application/json + +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [...] +} +``` + +### Authentication + +Support various authentication methods: + +#### API Key Authentication +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +#### OAuth2 Authentication +```json +{ + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token" + } +} +``` + +## Tool Consumer Implementation + +### Client Configuration + +Configure your UTCP client: + +#### File-based Configuration +```yaml +# utcp-config.yaml +manual_call_templates: + - name: weather_service + call_template_type: http + url: https://api.weather.com/utcp + +variables: + WEATHER_API_KEY: your-api-key + +load_variables_from: + - variable_loader_type: dotenv + env_file_path: .env +``` + +#### Programmatic Configuration +```json +{ + "manual_call_templates": [ + { + "name": "service_name", + "call_template_type": "http", + "url": "https://api.example.com/utcp" + } + ], + "variables": { + "API_KEY": "your-key" + } +} +``` + +### Tool Discovery + +Discover available tools: + +1. **List Tools**: Get all available tools +2. **Tool Information**: Get detailed tool metadata +3. **Filter Tools**: Find tools by name, tags, or description + +### Tool Execution + +Execute tools with proper error handling: + +1. **Basic Calls**: Simple tool invocation +2. **Batch Calls**: Execute multiple tools +3. **Context Passing**: Pass context between calls +4. **Error Handling**: Handle failures gracefully + +## Advanced Implementation Patterns + +### Custom Protocol Plugins + +Extend UTCP with custom communication protocols: + +1. **Define Call Template**: Structure for your protocol +2. **Implement Communication Handler**: Protocol-specific logic +3. **Register Protocol**: Make it available to clients + +Example custom protocol structure: +```json +{ + "call_template_type": "custom", + "custom_field": "value", + "timeout": 30 +} +``` + +### Custom Tool Repositories + +Implement custom tool storage: + +1. **Tool Storage**: How tools are stored and retrieved +2. **Search Functionality**: Tool discovery and filtering +3. **Caching**: Performance optimization +4. **Synchronization**: Multi-client coordination + +### Testing Strategies + +#### Unit Testing Tool Providers +- Test manual generation +- Validate tool definitions +- Test authentication +- Mock external dependencies + +#### Integration Testing +- Test tool discovery +- Test tool execution +- Test error scenarios +- Test performance + +#### End-to-End Testing +- Test complete workflows +- Test multiple protocols +- Test real-world scenarios +- Test scalability + +## Best Practices + +### Performance +- Use connection pooling +- Implement caching +- Optimize tool discovery +- Monitor response times + +### Security +- Validate all inputs +- Use secure authentication +- Implement rate limiting +- Log security events + +### Reliability +- Implement retry logic +- Handle network failures +- Use circuit breakers +- Monitor tool health + +### Maintainability +- Version your manuals +- Document all tools +- Use consistent naming +- Provide examples + +## Deployment Considerations + +### Scaling +- Load balance tool providers +- Cache tool discoveries +- Use async processing +- Monitor resource usage + +### Monitoring +- Track tool usage +- Monitor error rates +- Log performance metrics +- Set up alerts + +### Security +- Use HTTPS everywhere +- Implement proper authentication +- Validate all inputs +- Monitor for abuse + +## Language-Specific Implementation + +For detailed implementation examples and code samples in your programming language: + +- **Multi-language**: [UTCP Implementation Examples](https://github.com/universal-tool-calling-protocol) - Examples across Python, TypeScript, Go, and other languages +- **TypeScript**: [TypeScript UTCP Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/tree/main/docs) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) + +For more detailed information, see: +- [Communication Protocol Plugins](./protocols/index.md) +- [API Reference](./api/index.md) +- [Security Considerations](./security.md) diff --git a/docs/index.md b/docs/index.md index 4f0a4e4..2355a3e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,152 +4,219 @@ title: Introduction sidebar_position: 1 --- -# Introduction to UTCP 1.0 +# Universal Tool Calling Protocol (UTCP) -The Universal Tool Calling Protocol (UTCP) is a lightweight, secure, and scalable standard for defining and interacting with tools across a wide variety of communication protocols. Version 1.0 introduces a modular core with a plugin-based architecture, making it more extensible, testable, and easier to package. - -## Core Components +:::info Language Examples +UTCP is available in multiple languages - see [Python](https://github.com/universal-tool-calling-protocol/python-utcp), [TypeScript](https://github.com/universal-tool-calling-protocol/typescript-utcp), [Go](https://github.com/universal-tool-calling-protocol/go-utcp), and other implementations in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol). +::: -UTCP consists of four main components: +UTCP is a lightweight, secure, and scalable standard that enables AI agents and applications to discover and call tools directly using their native protocols - **no wrapper servers required**. -1. [**Manuals**](./api/core/utcp/data/utcp_manual.md): The standard tool provider description format that contains tool definitions -2. [**Tools**](./api/core/utcp/data/tool.md): The individual capabilities that can be called -3. [**Call Templates**](./api/core/utcp/data/call_template.md): The communication configurations that specify how tools are accessed. Concretely this maps a tool name and provided arguments to an actual API request in a communication protocol. -4. [**UtcpClient**](./api/core/utcp/utcp_client.md): The client that calls tools using the call templates. +## Why UTCP? -## The "Manual" Approach +### The Problem with Current Approaches +Most tool integration solutions force you to: +- Build and maintain wrapper servers for every tool +- Route all traffic through a middleman protocol +- Reimplement existing authentication and security +- Accept additional latency and complexity -UTCP's fundamental philosophy is to act as a descriptive manual rather than a prescriptive middleman: +### The UTCP Solution +UTCP acts as a **"manual"** that tells agents how to call your tools directly: -:::note -A UTCP Manual tells an agent: "Here is a tool. Here is its native endpoint (HTTP, WebSocket, CLI, etc.), and here is how to call it directly." +:::tip Core Philosophy +*"If a human can call your API, an AI agent should be able to call it too - with the same security and no additional infrastructure."* ::: -This approach eliminates the need for wrapper servers and allows direct communication between agents and tools. - -## New Architecture in 1.0 - -UTCP has been refactored into a core library and a set of optional plugins: - -### Core Package (`utcp`) -- **Data Models**: Pydantic models for [`Tool`](./api/core/utcp/data/tool.md), [`CallTemplate`](./api/core/utcp/data/call_template.md), [`UtcpManual`](./api/core/utcp/data/utcp_manual.md), and [`Auth`](./api/core/utcp/data/auth.md) -- **Pluggable Interfaces**: [`CommunicationProtocol`](./api/core/utcp/interfaces/communication_protocol.md), [`ConcurrentToolRepository`](./api/core/utcp/interfaces/concurrent_tool_repository.md), [`ToolSearchStrategy`](./api/core/utcp/interfaces/tool_search_strategy.md), [`VariableSubstitutor`](./api/core/utcp/interfaces/variable_substitutor.md), [`ToolPostProcessor`](./api/core/utcp/interfaces/tool_post_processor.md) -- **Default Implementations**: [`UtcpClient`](./api/core/utcp/utcp_client.md), [`InMemToolRepository`](./api/core/utcp/implementations/in_mem_tool_repository.md), [`TagAndDescriptionWordMatchStrategy`](./api/core/utcp/implementations/tag_search.md) - -### Protocol Plugins -- `utcp-http`: Supports HTTP, SSE, and streamable HTTP, plus an OpenAPI converter -- `utcp-cli`: For wrapping local command-line tools -- `utcp-mcp`: For interoperability with the Model Context Protocol (MCP) -- `utcp-text`: For reading text files -- `utcp-socket`: TCP and UDP protocols (work in progress) -- `utcp-gql`: GraphQL (work in progress) - -## Minimal Example - -Let's see how easy it is to use UTCP with a minimal example. - -### 1. Defining a Tool (Tool Provider) - -Create a simple HTTP endpoint that serves a UTCP Manual (JSON): - -```python -# app.py -from fastapi import FastAPI - -app = FastAPI() - -@app.get("/utcp") -def utcp_discovery(): - return { - "manual_version": "1.0.0", - "utcp_version": "1.0.1", - "tools": [ - { - "name": "get_weather", - "description": "Get current weather for a location", - "inputs": { - "type": "object", - "properties": { - "location": { - "type": "string", - "description": "City name" - } - }, - "required": ["location"] - }, - "outputs": { - "type": "object", - "properties": { - "temperature": {"type": "number"}, - "conditions": {"type": "string"} - } - }, - "tool_call_template": { - "call_template_type": "http", - "url": "https://example.com/api/weather", - "http_method": "GET" - } - } - ] - } +## Quick Start (5 Minutes) + +### 1. Install UTCP + +```bash +# Example installation (Python) +pip install utcp utcp-http + +# Example installation (Node.js) +npm install @utcp/core @utcp/http -# Implement the actual weather API endpoint -@app.get("/api/weather") -def get_weather(location: str): - # In a real app, you'd fetch actual weather data - return {"temperature": 22.5, "conditions": "Sunny"} +# See language-specific documentation for other implementations ``` -Run the server: +### 2. Expose Your First Tool + +Add a discovery endpoint to your existing API: + +**Add endpoint**: `GET /utcp` +**Return your UTCP manual**: +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [{ + "name": "get_weather", + "description": "Get current weather for a location", + "inputs": { + "type": "object", + "properties": {"location": {"type": "string"}}, + "required": ["location"] + }, + "tool_call_template": { + "call_template_type": "http", + "url": "http://localhost:8000/weather", + "http_method": "GET" + } + }], + "auth": { + "auth_type": "api_key", + "api_key": "${WEATHER_API_KEY}", + "var_name": "appid", + "location": "query" + } + } + }] +} +``` -```bash -uvicorn app:app --reload +### 3. Call Your Tool + +**Configure UTCP client**: +```json +{ + "manual_call_templates": [{ + "name": "weather_api", + "call_template_type": "http", + "url": "http://localhost:8000/utcp", + "http_method": "GET" + }] +} ``` -### 2. Using the Tool (Client) - -```python -# client.py -import asyncio -from utcp.utcp_client import UtcpClient -from utcp_http.http_call_template import HttpCallTemplate - -async def main(): - # Create a UTCP client with configuration - client = await UtcpClient.create(config={ - "manual_call_templates": [ - { - "name": "weather_service", - "call_template_type": "http", - "http_method": "GET", - "url": "http://localhost:8000/utcp" - } - ] - }) - - # Tools are automatically registered from the manual call templates - # Call a tool by its namespaced name: {manual_name}.{tool_name} - result = await client.call_tool( - "weather_service.get_weather", - tool_args={"location": "San Francisco"} - ) - print(f"Weather: {result['temperature']}°C, {result['conditions']}") - -if __name__ == "__main__": - asyncio.run(main()) +**Call the tool**: +1. Initialize UTCP client with configuration +2. Discover tools from weather API +3. Call `get_weather` tool with location parameter +4. Receive weather data response ``` -Run the client: +**That's it!** Your tool is now discoverable and callable by any UTCP client. -```bash -python client.py +## Key Benefits + +| Benefit | Description | +|---------|-------------| +| **🚀 Zero Latency Overhead** | Direct tool calls, no proxy servers | +| **🔒 Native Security** | Use your existing authentication and authorization | +| **🌐 Protocol Flexibility** | HTTP, WebSocket, CLI, GraphQL, and more | +| **⚡ Easy Integration** | Add one endpoint, no infrastructure changes | +| **📈 Scalable** | Leverage your existing scaling and monitoring | + +## How It Works + +```mermaid +graph LR + A[AI Agent] -->|1. Discover| B[UTCP Manual] + B -->|2. Learn| C[Tool Definitions] + A -->|3. Call Directly| D[Your API] + D -->|4. Response| A ``` -## Benefits of the UTCP Approach +1. **Discovery**: Agent fetches your UTCP manual +2. **Learning**: Agent understands how to call your tools +3. **Direct Calling**: Agent calls your API directly using native protocols +4. **Response**: Your API responds normally + +## Supported Protocols + +UTCP supports multiple communication protocols through plugins: + +| Protocol | Use Case | Plugin | Status | +|----------|----------|--------|--------| +| **[HTTP](./protocols/http.md)** | REST APIs, webhooks | `utcp-http` | ✅ Stable | +| **[WebSocket](./protocols/websocket.md)** | Real-time communication | `utcp-websocket` | ✅ Stable | +| **[CLI](./protocols/cli.md)** | Command-line tools | `utcp-cli` | ✅ Stable | +| **[Server-Sent Events](./protocols/streamable-http.md)** | Streaming data | `utcp-http` | ✅ Stable | +| **[Text Files](./protocols/text.md)** | File reading | `utcp-text` | ✅ Stable | +| **[MCP](./protocols/mcp.md)** | MCP interoperability | `utcp-mcp` | ✅ Stable | + +[View all protocols →](./protocols/index.md) + +## Architecture Overview -1. **Direct Communication**: The client calls the tool's native endpoint directly -2. **No Wrapper Infrastructure**: No need to build and maintain wrapper servers -3. **Leverage Existing Systems**: Uses the tool's existing authentication, rate limiting, etc. -4. **Flexible Protocol Support**: Works with any communication protocol (HTTP, WebSockets, CLI, etc.) +UTCP v1.0 features a modular, plugin-based architecture: + +### Core Components +- **[Manuals](./api/core/utcp/data/utcp_manual.md)**: Tool definitions and metadata +- **[Tools](./api/core/utcp/data/tool.md)**: Individual callable capabilities +- **[Call Templates](./api/core/utcp/data/call_template.md)**: Protocol-specific call instructions +- **[UTCP Client](./api/core/utcp/utcp_client.md)**: Tool discovery and execution engine + +### Plugin System +- **Protocol Plugins**: HTTP, WebSocket, CLI, etc. +- **Custom Protocols**: Extend with your own communication methods +- **Tool Repositories**: Pluggable storage for tool definitions +- **Search Strategies**: Customizable tool discovery algorithms + +[Learn more about the architecture →](./api/index.md) + +## Who Should Use UTCP? + +### 🛠️ Tool Providers +You have APIs, services, or tools that you want AI agents to use: +- **Existing API owners** - Expose your REST APIs to AI agents +- **SaaS providers** - Make your services AI-accessible +- **Enterprise teams** - Enable internal tool usage by AI systems + +[**Get started as a tool provider →**](./for-tool-providers.md) + +### 🤖 Tool Consumers +You're building AI agents or applications that need to call external tools: +- **AI agent developers** - Give your agents access to external capabilities +- **Application builders** - Integrate third-party tools seamlessly +- **Enterprise developers** - Connect to internal and external services + +[**Get started as a tool consumer →**](./implementation.md) + +## UTCP vs Alternatives + +| Feature | UTCP | MCP | Custom Wrappers | +|---------|------|-----|-----------------| +| **Infrastructure** | None required | Wrapper servers | Custom servers | +| **Latency** | Direct calls | Double hop | Variable | +| **Security** | Native | Reimplemented | Custom | +| **Protocols** | Multiple | HTTP streaming | Single | +| **Maintenance** | Minimal | High | Very high | + +[**Detailed comparison with MCP →**](./utcp-vs-mcp.md) + +## Next Steps + +### For Tool Providers +1. **[Read the provider guide](./for-tool-providers.md)** - Learn how to expose your tools +2. **[Choose your protocol](./protocols/index.md)** - Select the right communication method +3. **[Implement your manual](./implementation.md)** - Add UTCP to your existing API +4. **[Secure your tools](./security.md)** - Implement proper authentication + +### For Tool Consumers +1. **[Read the implementation guide](./implementation.md)** - Learn how to build UTCP clients +2. **[Explore protocols](./protocols/index.md)** - Understand available communication options +3. **[Check examples](https://github.com/universal-tool-calling-protocol)** - See real-world implementations +4. **[Join the community](https://discord.gg/ZpMbQ8jRbD)** - Get help and share experiences + +### Migration from Other Systems +- **[From UTCP v0.1](./migration-v0.1-to-v1.0.md)** - Upgrade to the latest version +- **[From MCP](./protocols/mcp.md)** - Migrate from Model Context Protocol +- **[From custom solutions](./implementation.md)** - Replace existing tool integrations + +## Community & Support + +- **[GitHub Organization](https://github.com/universal-tool-calling-protocol)** - Source code and issues +- **[Discord Community](https://discord.gg/ZpMbQ8jRbD)** - Real-time help and discussions +- **[Tool Registry](https://utcp.io/registry)** - Discover available tools +- **[RFC Process](/about/RFC)** - Contribute to the specification + +--- -In the following sections, we'll explore the details of UTCP's components and how to implement them in your applications. +**Ready to get started?** Choose your path: +- 🛠️ [**I want to expose my tools**](./for-tool-providers.md) +- 🤖 [**I want to call tools**](./implementation.md) +- 📚 [**I want to learn more**](./protocols/index.md) diff --git a/docs/migration-v0.1-to-v1.0.md b/docs/migration-v0.1-to-v1.0.md new file mode 100644 index 0000000..eb50de7 --- /dev/null +++ b/docs/migration-v0.1-to-v1.0.md @@ -0,0 +1,465 @@ +--- +id: migration-v0.1-to-v1.0 +title: Migration Guide - v0.1 to v1.0 +sidebar_position: 7 +--- + +# Migration Guide: v0.1 to v1.0 + +This guide helps you migrate from UTCP v0.1 to v1.0, which introduces significant architectural improvements and new features. + +## Overview of Changes + +### Major Changes in v1.0 + +1. **Plugin Architecture**: Core functionality split into pluggable components +2. **Improved Data Models**: Enhanced Pydantic models with better validation +3. **New Protocol Support**: Additional communication protocols +4. **Better Error Handling**: More specific exception types +5. **Enhanced Authentication**: Expanded authentication options +6. **Performance Improvements**: Optimized client and protocol implementations + +### Breaking Changes + +| Component | v0.1 | v1.0 | Impact | +|-----------|------|------|--------| +| **Package Structure** | Single package | Core + plugins | High | +| **Client API** | `UtcpClient()` | `UtcpClient.create()` | Medium | +| **Configuration** | Simple dict | Structured config | Medium | +| **Protocol Registration** | Automatic | Plugin-based | High | +| **Error Types** | Generic exceptions | Specific exception types | Low | + +## Installation Changes + +### v0.1 Installation + +```bash +pip install utcp +``` + +### v1.0 Installation + +```bash +# Core package +pip install utcp + +# Protocol plugins (install as needed) +pip install utcp-http utcp-cli utcp-websocket utcp-text utcp-mcp +``` + +## Client Migration + +### v0.1 Client Code + +**Legacy v0.1 client configuration:** +- Import UTCP client library +- Configure providers with provider-specific settings (name, type, URL, HTTP method) +- Call tools using provider.tool_name format +- Synchronous tool calling interface + +### v1.0 Client Code + +**New v1.0 client configuration:** +- Import updated UTCP client library from new module path +- Use async factory method for client creation +- Configure manual call templates instead of providers +- Use async/await pattern for tool calling +- Enhanced error handling and response processing + +## Configuration Migration + +### v0.1 Configuration + +```yaml +providers: + - name: weather_service + provider_type: http + url: https://weather.example.com/utcp + http_method: GET + - name: file_service + provider_type: cli + command: cat + args: ["${filename}"] + +variables: + API_KEY: your_api_key +``` + +### v1.0 Configuration + +```yaml +manual_call_templates: + - name: weather_service + call_template_type: http + url: https://weather.example.com/utcp + http_method: GET + - name: file_service + call_template_type: cli + command: cat + args: ["${filename}"] + +load_variables_from: + - variable_loader_type: dotenv + env_file_path: .env +``` + +## Manual Format Migration + +### v0.1 Manual Format + +```json +{ + "utcp_version": "0.1.0", + "provider_info": { + "name": "weather_api", + "version": "1.0.0" + }, + "tools": [ + { + "name": "get_weather", + "description": "Get weather data", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"} + } + }, + "provider": { + "provider_type": "http", + "url": "https://api.weather.com/current", + "method": "GET" + } + } + ] +} +``` + +### v1.0 Manual Format + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "info": { + "title": "Weather API", + "version": "1.0.0", + "description": "Weather data and forecasting tools" + }, + "tools": [ + { + "name": "get_weather", + "description": "Get weather data", + "inputs": { + "type": "object", + "properties": { + "location": {"type": "string"} + }, + "required": ["location"] + }, + "outputs": { + "type": "object", + "properties": { + "temperature": {"type": "number"}, + "conditions": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.weather.com/current", + "http_method": "GET", + }, + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "appid", + "location": "query" + } + } + } + ] +} +``` + +## Protocol Migration + +### HTTP Protocol Changes + +#### v0.1 HTTP Provider + +```json +{ + "provider_type": "http", + "url": "https://api.example.com/endpoint", + "method": "POST", + "headers": {"Authorization": "Bearer ${TOKEN}"}, + "body": {"data": "${input}"} +} +``` + +#### v1.0 HTTP Call Template + +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "POST", + "headers": {"Authorization": "Bearer ${TOKEN}"}, + "body_field": "body", + "auth": { + "auth_type": "api_key", + "api_key": "${TOKEN}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +### CLI Protocol Changes + +#### v0.1 CLI Provider + +```json +{ + "provider_type": "cli", + "command": "python", + "args": ["script.py", "${input}"], + "cwd": "/app" +} +``` + +#### v1.0 CLI Call Template + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "cd /app", + "append_to_final_output": false + }, + { + "command": "python script.py UTCP_ARG_input_UTCP_END", + "append_to_final_output": true + } + ], + "working_dir": "/app", + "env_vars": { + "PYTHONPATH": "/app/lib" + } +} +``` + +## Authentication Migration + +### v0.1 Authentication + +```json +{ + "provider": { + "provider_type": "http", + "url": "https://api.example.com/data", + "headers": { + "Authorization": "Bearer ${API_TOKEN}" + } + } +} +``` + +### v1.0 Authentication + +```json +{ + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/data", + "auth": { + "auth_type": "api_key", + "api_key": "${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + } + } +} +``` + +## Error Handling Migration + +### v0.1 Error Handling + +**Basic error handling in v0.1:** +- Simple try/catch block with generic Exception handling +- Limited error information and context +- Basic error message printing + +### v1.0 Error Handling + +**Enhanced error handling in v1.0:** +- Import specific exception types from utcp.exceptions module +- Handle ToolNotFoundError for missing tools +- Handle AuthenticationError for auth failures +- Handle ToolCallError for tool execution failures +- Handle base UtcpError for general UTCP errors +- Use try/catch blocks with specific exception handling + +## Step-by-Step Migration + +### Step 1: Update Dependencies + +```bash +# Uninstall old version +pip uninstall utcp + +# Install new version with plugins +pip install utcp utcp-http utcp-cli utcp-websocket utcp-text +``` + +### Step 2: Update Client Code + +**Migration to async pattern:** +- **Before**: Synchronous client creation and tool calls +- **After**: Async client factory method and async tool calls +- Import asyncio module for async execution +- Use async/await keywords for client creation and tool calls +- Run async main function with asyncio.run() + +### Step 3: Update Configuration + +**Configuration migration helper:** +- Create migration function to convert v0.1 config to v1.0 format +- Transform providers array to manual_call_templates array +- Add variable_loaders configuration for environment variables +- Map provider_type to call_template_type +- Migrate HTTP provider settings (URL, method, headers, body) +- Migrate CLI provider settings (command, args, working_directory) +- Load old configuration and apply migration helper +- Create new client with migrated configuration + +### Step 4: Update Manual Format + +**Manual migration helper:** +- Create migration function to convert v0.1 manual to v1.0 format +- Update manual_version and utcp_version fields +- Transform provider_info to info structure (title, version, description) +- Migrate tools array with updated structure +- Convert tool parameters to inputs field +- Transform provider configuration to tool_call_template +- Map provider_type to call_template_type +- Migrate HTTP provider settings (URL, method, headers, body) + +### Step 5: Test Migration + +**Testing migrated client:** +- Import pytest and UtcpClient for async testing +- Create test function with pytest.mark.asyncio decorator +- Test client creation with migrated configuration +- Test tool discovery functionality (list_tools) +- Test tool calling with sample parameters +- Assert expected results and functionality + +## Common Migration Issues + +### Issue 1: Async/Await + +**Problem**: v1.0 client methods are async +**Solution**: Add `async`/`await` keywords + +**Code changes:** +- **Before**: Synchronous tool calling (result = client.call_tool("tool", args)) +- **After**: Async tool calling (result = await client.call_tool("tool", args)) + +### Issue 2: Configuration Format + +**Problem**: Configuration structure changed +**Solution**: Use migration helper or update manually + +**Configuration changes:** +- **Before**: Use "providers" array in configuration +- **After**: Use "manual_call_templates" array in configuration + +### Issue 3: Plugin Dependencies + +**Problem**: Protocol implementations not found +**Solution**: Install required plugins + +```bash +pip install utcp-http utcp-cli utcp-websocket +``` + +### Issue 4: Manual Format + +**Problem**: Old manual format not recognized +**Solution**: Update manual structure + +```json +// Before +{"provider": {"provider_type": "http"}} + +// After +{"tool_call_template": {"call_template_type": "http"}} +``` + +## Validation Tools + +### Configuration Validator + +**Configuration validation helper:** +- Import UtcpClientConfig from utcp.data.utcp_client_config +- Create validation function that accepts configuration dictionary +- Use UtcpClientConfig constructor to validate configuration structure +- Handle validation exceptions and provide error messages +- Return validated config object or None on failure + +### Manual Validator + +**Manual validation helper:** +- Import UtcpManual from utcp.data.utcp_manual +- Create validation function that accepts manual dictionary +- Use UtcpManual constructor to validate manual structure +- Handle validation exceptions and provide error messages +- Return validated manual object or None on failure + +## Best Practices for Migration + +1. **Gradual Migration**: Migrate one component at a time +2. **Test Thoroughly**: Test each migrated component +3. **Backup Configurations**: Keep backups of v0.1 configurations +4. **Use Validation**: Validate configurations and manuals +5. **Monitor Performance**: Check for performance regressions +6. **Update Documentation**: Update internal documentation +7. **Train Team**: Ensure team understands new patterns + +## Post-Migration Checklist + +- [ ] All dependencies updated +- [ ] Client code uses async/await +- [ ] Configuration format updated +- [ ] Manual format updated +- [ ] Error handling updated +- [ ] Tests passing +- [ ] Performance acceptable +- [ ] Documentation updated +- [ ] Team trained on changes + +## Getting Help + +If you encounter issues during migration: + +1. **Check Documentation**: Review the [Implementation Guide](./implementation.md) +2. **GitHub Issues**: Search existing issues or create new ones +3. **Discord Community**: Join the [UTCP Discord](https://discord.gg/ZpMbQ8jRbD) +4. **Examples**: Check the [examples repository](https://github.com/universal-tool-calling-protocol) for implementations across multiple languages + +## Rollback Plan + +If migration issues occur, you can rollback: + +```bash +# Rollback to v0.1 +pip uninstall utcp utcp-http utcp-cli utcp-websocket utcp-text +pip install utcp==0.1.0 + +# Restore old configuration files +cp config-v0.1-backup.yaml config.yaml +``` + +Remember to test the rollback process in a non-production environment first. diff --git a/docs/protocols/cli.md b/docs/protocols/cli.md new file mode 100644 index 0000000..f0ac213 --- /dev/null +++ b/docs/protocols/cli.md @@ -0,0 +1,471 @@ +--- +id: cli +title: CLI Protocol +sidebar_position: 4 +--- + +# CLI Protocol + +The CLI protocol plugin (`utcp-cli`) enables UTCP to execute multi-command workflows and scripts. This protocol supports sequential command execution with state preservation and cross-platform compatibility. + +## Installation + +```bash +# Example installation (Python) +pip install utcp-cli + +# Example installation (Node.js) +npm install @utcp/cli +``` + +## Key Features + +- **Multi-Command Execution**: Execute multiple commands sequentially in a single subprocess +- **State Preservation**: Directory changes and environment persist between commands +- **Cross-Platform Script Generation**: PowerShell on Windows, Bash on Unix/Linux/macOS +- **Flexible Output Control**: Choose which command outputs to include in final result +- **Argument Substitution**: `UTCP_ARG_argname_UTCP_END` placeholder system +- **Output Referencing**: Access previous command outputs with `$CMD_0_OUTPUT`, `$CMD_1_OUTPUT` +- **Environment Variables**: Secure credential and configuration passing + +## Call Template Structure + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_target_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "git status --porcelain", + "append_to_final_output": false + }, + { + "command": "echo \"Status: $CMD_1_OUTPUT, Files: $(echo \"$CMD_1_OUTPUT\" | wc -l)\"", + "append_to_final_output": true + } + ], + "working_dir": "/tmp", + "env_vars": { + "GIT_AUTHOR_NAME": "UTCP Bot" + } +} +``` + +## Configuration Options + +### Required Fields + +- **`call_template_type`**: Must be `"cli"` +- **`commands`**: Array of `CommandStep` objects defining the sequence of commands + +### Optional Fields + +- **`working_dir`**: Working directory for command execution +- **`env_vars`**: Environment variables to set for all commands + +### CommandStep Object + +- **`command`**: Command string with `UTCP_ARG_argname_UTCP_END` placeholders +- **`append_to_final_output`**: Whether to include this command's output in the final result (defaults to `false` for all except the last command) + +For complete field specifications and validation rules, see the [CLI Call Template API Reference](../api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md). + +## Argument Substitution + +Use `UTCP_ARG_argname_UTCP_END` placeholders in command strings: + +```json +{ + "command": "git clone UTCP_ARG_repo_url_UTCP_END UTCP_ARG_target_dir_UTCP_END" +} +``` + +## Output Referencing + +Reference previous command outputs using `$CMD_N_OUTPUT` variables: + +```json +{ + "commands": [ + { + "command": "git status --porcelain", + "append_to_final_output": false + }, + { + "command": "echo \"Changes detected: $CMD_0_OUTPUT\"", + "append_to_final_output": true + } + ] +} +``` + +## Security Considerations + +:::danger Security Warning +CLI protocol executes commands on the local system. Always validate inputs and use with caution. +::: + +### Input Validation + +Always validate and sanitize inputs to prevent command injection: + +```json +{ + "name": "safe_file_read", + "description": "Safely read a file", + "inputs": { + "type": "object", + "properties": { + "filename": { + "type": "string", + "pattern": "^[a-zA-Z0-9._-]+$" + } + }, + "required": ["filename"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "cd /safe/directory", + "append_to_final_output": false + }, + { + "command": "cat UTCP_ARG_filename_UTCP_END", + "append_to_final_output": true + } + ], + "working_dir": "/safe/directory" + } +} +``` + +### Sandboxing + +Consider running CLI tools in sandboxed environments: + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "docker run --rm --read-only -v /safe/data:/data:ro alpine:latest cat /data/UTCP_ARG_filename_UTCP_END", + "append_to_final_output": true + } + ] +} +``` + +## Examples + +### Simple Command Execution + +```json +{ + "name": "get_system_info", + "description": "Get system information", + "inputs": { + "type": "object", + "properties": {} + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "uname -a", + "append_to_final_output": true + } + ] + } +} +``` + +### Multi-Step File Analysis + +```json +{ + "name": "analyze_directory", + "description": "Analyze files in a directory", + "inputs": { + "type": "object", + "properties": { + "path": { + "type": "string", + "description": "Directory path to analyze" + } + }, + "required": ["path"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_path_UTCP_END", + "append_to_final_output": false + }, + { + "command": "find . -type f | wc -l", + "append_to_final_output": false + }, + { + "command": "du -sh .", + "append_to_final_output": false + }, + { + "command": "echo \"Directory Analysis: $CMD_2_OUTPUT total size, $CMD_1_OUTPUT files\"", + "append_to_final_output": true + } + ] + } +} +``` + +### Git Workflow + +```json +{ + "name": "git_analysis", + "description": "Analyze git repository", + "inputs": { + "type": "object", + "properties": { + "repo_url": {"type": "string"}, + "target_dir": {"type": "string"} + }, + "required": ["repo_url", "target_dir"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "git clone UTCP_ARG_repo_url_UTCP_END UTCP_ARG_target_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "cd UTCP_ARG_target_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "git log --oneline -10", + "append_to_final_output": true + }, + { + "command": "find . -name '*.py' | wc -l", + "append_to_final_output": false + }, + { + "command": "echo \"Repository has $CMD_3_OUTPUT Python files\"", + "append_to_final_output": true + } + ], + "env_vars": { + "GIT_AUTHOR_NAME": "UTCP Bot", + "GIT_AUTHOR_EMAIL": "bot@utcp.dev" + } + } +} +``` + +### Python Development Pipeline + +```json +{ + "name": "python_pipeline", + "description": "Run Python development pipeline", + "inputs": { + "type": "object", + "properties": { + "project_dir": {"type": "string"}, + "script_name": {"type": "string"} + }, + "required": ["project_dir", "script_name"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_project_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "python -m pip install -r requirements.txt", + "append_to_final_output": false + }, + { + "command": "python UTCP_ARG_script_name_UTCP_END", + "append_to_final_output": true + } + ], + "working_dir": "/workspace", + "env_vars": { + "PYTHONPATH": "/workspace/lib", + "VIRTUAL_ENV": "/workspace/venv" + } + } +} +``` + +## Output Handling + +The CLI protocol executes all commands in a single subprocess and returns the combined output based on `append_to_final_output` settings: + +- Commands with `append_to_final_output: true` contribute to the final result +- Commands with `append_to_final_output: false` are executed for side effects only +- If no `append_to_final_output` is specified, only the last command's output is returned +- JSON output is automatically parsed if detected + +### Success Response + +```json +"Directory Analysis: 2.1G total size, 1247 files" +``` + +### JSON Output Detection + +If output starts with `{` or `[`, it's automatically parsed as JSON: + +```json +{ + "files": 1247, + "size": "2.1G", + "analysis_time": "2023-12-01T10:30:00Z" +} +``` + +## Environment Variables + +Set environment variables for all commands: + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "node app.js", + "append_to_final_output": true + } + ], + "env_vars": { + "NODE_ENV": "production", + "API_KEY": "${API_KEY}", + "PORT": "3000" + } +} +``` + +## Working Directory + +Specify the initial working directory: + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "make build", + "append_to_final_output": true + } + ], + "working_dir": "/project/src" +} +``` + +## Cross-Platform Considerations + +### Command Syntax + +Commands should use appropriate syntax for the target platform: + +**Windows (PowerShell):** +```json +{ + "commands": [ + {"command": "Get-ChildItem UTCP_ARG_path_UTCP_END"}, + {"command": "Set-Location UTCP_ARG_dir_UTCP_END"} + ] +} +``` + +**Unix/Linux/macOS (Bash):** +```json +{ + "commands": [ + {"command": "ls -la UTCP_ARG_path_UTCP_END"}, + {"command": "cd UTCP_ARG_dir_UTCP_END"} + ] +} +``` + +### Universal Commands + +Some commands work across platforms: +```json +{ + "commands": [ + {"command": "git status"}, + {"command": "python --version"}, + {"command": "node -v"} + ] +} +``` + +## Best Practices + +1. **Validate Inputs**: Always validate and sanitize user inputs using JSON Schema patterns +2. **Use Absolute Paths**: Prefer absolute paths for commands and files when possible +3. **Control Output**: Use `append_to_final_output` to control which command outputs are returned +4. **Reference Previous Output**: Use `$CMD_N_OUTPUT` to reference previous command results +5. **Limit Permissions**: Run with minimal necessary permissions +6. **Sandbox Execution**: Use containers or chroot when possible +7. **Handle Cross-Platform**: Consider platform-specific command syntax +8. **Environment Variables**: Use `env_vars` for secure credential passing + +## Common Use Cases + +- **Multi-step Builds**: setup → compile → test → package +- **Git Workflows**: clone → analyze → commit → push +- **Data Pipelines**: fetch → transform → validate → output +- **File Operations**: navigate → search → process → report +- **Development Tools**: install dependencies → run tests → generate docs +- **System Administration**: check status → backup → cleanup → verify + +## Error Handling + +| Error Type | Description | Handling | +|------------|-------------|---------| +| Missing Arguments | Required UTCP_ARG placeholder not provided | Validation error | +| Command Not Found | Command doesn't exist | Script execution error | +| Permission Denied | Insufficient permissions | Script execution error | +| Timeout | Script exceeded timeout | Async timeout error | +| Non-zero Exit | Script failed | Return stderr output | + +## Testing CLI Tools + +```python +import pytest +from utcp.utcp_client import UtcpClient + +@pytest.mark.asyncio +async def test_multi_command_cli_tool(): + client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "test_cli", + "call_template_type": "cli", + "commands": [ + { + "command": "echo UTCP_ARG_message_UTCP_END", + "append_to_final_output": false + }, + { + "command": "echo \"Previous: $CMD_0_OUTPUT\"" + } + ] + }] + }) + + result = await client.call_tool("test_cli.echo_chain", {"message": "hello"}) + assert "Previous: hello" in result +``` diff --git a/docs/protocols/http.md b/docs/protocols/http.md new file mode 100644 index 0000000..6340aeb --- /dev/null +++ b/docs/protocols/http.md @@ -0,0 +1,236 @@ +--- +id: http +title: HTTP Protocol +sidebar_position: 1 +--- + +# HTTP Protocol Plugin + +The HTTP protocol plugin enables UTCP to call tools via HTTP/HTTPS requests. This is the most commonly used protocol for REST APIs, webhooks, and web services. + +## Call Template Structure + +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "GET|POST|PUT|DELETE|PATCH", + "headers": { + "Content-Type": "application/json", + "User-Agent": "UTCP-Client/1.0" + }, + "query_params": { + "param1": "${variable1}", + "param2": "static_value" + }, + "body": { + "data": "${input_data}", + "timestamp": "${current_time}" + }, + "timeout": 30, + "verify_ssl": true, + "auth": { + "auth_type": "api_key|basic|oauth2", + "api_key": "${API_KEY}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +## Supported HTTP Methods + +| Method | Use Case | Body Support | +|--------|----------|--------------| +| `GET` | Retrieve data | No | +| `POST` | Create resources, submit data | Yes | +| `PUT` | Update/replace resources | Yes | +| `PATCH` | Partial updates | Yes | +| `DELETE` | Remove resources | Optional | + +## Authentication Methods + +### API Key Authentication +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +### Basic Authentication +```json +{ + "auth": { + "auth_type": "basic", + "username": "${USERNAME}", + "password": "${PASSWORD}" + } +} +``` + +### OAuth2 Authentication +```json +{ + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "read write" + } +} +``` + +## Variable Substitution + +Variables in call templates are substituted with values from: +- Tool call arguments: `${argument_name}` +- Environment variables: `${ENV_VAR}` +- Configuration variables: `${config.variable}` + +Example: +```json +{ + "url": "https://api.example.com/users/${user_id}", + "headers": { + "Authorization": "Bearer ${ACCESS_TOKEN}" + }, + "query_params": { + "format": "${output_format}", + "limit": "${max_results}" + } +} +``` + +## OpenAPI Integration + +The HTTP protocol plugin can automatically generate UTCP manuals from OpenAPI/Swagger specifications: + +1. **Automatic Discovery**: Point to an OpenAPI spec URL +2. **Schema Conversion**: Converts OpenAPI paths to UTCP tools +3. **Authentication Mapping**: Maps OpenAPI security schemes to UTCP auth +4. **Parameter Mapping**: Converts OpenAPI parameters to UTCP inputs + +Example OpenAPI to UTCP conversion: +```yaml +# OpenAPI Specification +paths: + /users/{id}: + get: + parameters: + - name: id + in: path + required: true + schema: + type: string +``` + +Becomes: +```json +{ + "name": "get_user", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users/${id}", + "http_method": "GET" + }, + "inputs": { + "type": "object", + "properties": { + "id": {"type": "string"} + }, + "required": ["id"] + } +} +``` + +## Response Handling + +HTTP responses are processed based on: +- **Status Codes**: 2xx considered success, others as errors +- **Content-Type**: JSON, XML, text, and binary content support +- **Headers**: Response headers available in tool output +- **Error Mapping**: HTTP errors mapped to UTCP exceptions + +## Configuration Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `timeout` | number | 30 | Request timeout in seconds | +| `verify_ssl` | boolean | true | Verify SSL certificates | +| `follow_redirects` | boolean | true | Follow HTTP redirects | +| `max_redirects` | number | 5 | Maximum redirect hops | +| `retry_count` | number | 0 | Number of retry attempts | +| `retry_delay` | number | 1 | Delay between retries (seconds) | + +## Security Considerations + +### SSL/TLS Verification +- Always verify SSL certificates in production +- Use `verify_ssl: false` only for testing/development +- Consider certificate pinning for high-security applications + +### Authentication Security +- Store credentials in environment variables, not in configuration files +- Use OAuth2 for user-facing applications +- Rotate API keys regularly +- Implement proper token refresh for OAuth2 + +### Input Validation +- Validate all input parameters before substitution +- Sanitize user inputs to prevent injection attacks +- Use allowlists for acceptable parameter values +- Implement rate limiting on the tool provider side + +### Network Security +- Use HTTPS for all production communications +- Implement proper firewall rules +- Consider using VPNs or private networks for sensitive tools +- Monitor and log all HTTP requests for security analysis + +## Error Handling + +Common HTTP errors and their meanings: + +| Status Code | Error Type | Description | +|-------------|------------|-------------| +| 400 | Bad Request | Invalid request parameters | +| 401 | Unauthorized | Authentication required or failed | +| 403 | Forbidden | Access denied | +| 404 | Not Found | Resource doesn't exist | +| 429 | Rate Limited | Too many requests | +| 500 | Server Error | Internal server error | +| 503 | Service Unavailable | Service temporarily unavailable | + +## Best Practices + +### Performance +- Use connection pooling for multiple requests +- Implement appropriate timeouts +- Consider request/response compression +- Cache responses when appropriate + +### Reliability +- Implement retry logic with exponential backoff +- Handle network failures gracefully +- Use circuit breakers for unreliable services +- Monitor response times and error rates + +### Maintainability +- Use descriptive tool names and descriptions +- Document all required parameters +- Provide usage examples +- Version your APIs and update call templates accordingly + +## Language-Specific Implementation + +For implementation details and examples in your programming language: + +- **Multi-language**: [UTCP HTTP Protocol Examples](https://github.com/universal-tool-calling-protocol) - HTTP protocol examples across multiple languages +- **TypeScript**: [TypeScript HTTP Protocol Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/blob/main/docs/protocols/http.md) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) diff --git a/docs/protocols/index.md b/docs/protocols/index.md new file mode 100644 index 0000000..9a4f701 --- /dev/null +++ b/docs/protocols/index.md @@ -0,0 +1,93 @@ +--- +id: protocols +title: Protocols +sidebar_position: 4 +--- + +# Communication Protocol Plugins + +UTCP v1.0 features a modular, plugin-based architecture where different communication protocols are implemented as separate plugins. Each protocol plugin provides call templates and communication handlers for specific transport methods. + +## Available Protocol Plugins + +| Protocol | Plugin Package | Call Template | Use Cases | +|----------|----------------|---------------|-----------| +| **[HTTP](./http.md)** | `utcp-http` | `HttpCallTemplate` | REST APIs, webhooks, web services | +| **[WebSocket](./websocket.md)** | `utcp-websocket` | `WebSocketCallTemplate` | Real-time communication, streaming | +| **[CLI](./cli.md)** | `utcp-cli` | `CliCallTemplate` | Command-line tools, scripts | +| **[Server-Sent Events](./sse.md)** | `utcp-http` | `SseCallTemplate` | Event streaming, live updates | +| **[Text Files](./text.md)** | `utcp-text` | `TextCallTemplate` | File reading, static content | +| **[MCP](./mcp.md)** | `utcp-mcp` | `McpCallTemplate` | Model Context Protocol interop | + +## Plugin Architecture + +Each protocol plugin consists of: + +### Call Templates +Define how to structure calls for the specific protocol: +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "POST", + "headers": {"Content-Type": "application/json"} +} +``` + +### Communication Protocols +Handle the actual communication logic for each protocol type. The implementation varies by programming language. + +## Installing Protocol Plugins + +Protocol plugins are available for different programming languages: + +```bash +# Example installation (Python) +pip install utcp-http utcp-cli utcp-websocket utcp-text utcp-mcp + +# Example installation (Node.js) +npm install @utcp/http @utcp/cli @utcp/websocket @utcp/text @utcp/mcp +``` + +For other languages, check the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) + +## Creating Custom Protocol Plugins + +You can extend UTCP with custom communication protocols by implementing the protocol interface in your chosen language. Each implementation must: + +1. **Define Call Templates**: Specify the structure for protocol-specific calls +2. **Implement Communication Handler**: Handle the actual protocol communication +3. **Register the Protocol**: Make it available to the UTCP client + +Example call template structure: +```json +{ + "call_template_type": "custom", + "custom_field": "value", + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}" + } +} +``` + +## Protocol Selection Guide + +Choose the right protocol plugin based on your needs: + +- **HTTP**: Most common for REST APIs and web services +- **WebSocket**: Real-time bidirectional communication +- **CLI**: Wrapping existing command-line tools +- **SSE**: Server-sent events for streaming data +- **Text**: Reading configuration files or static content +- **MCP**: Interoperability with Model Context Protocol tools + +## Language-Specific Documentation + +For implementation details and examples in your programming language: + +- **Multi-language**: [UTCP Implementation Examples](https://github.com/universal-tool-calling-protocol) - Examples across Python, TypeScript, Go, and other languages +- **TypeScript**: [TypeScript UTCP Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/tree/main/docs) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) + +For detailed information about each protocol plugin, see the individual protocol documentation pages. diff --git a/docs/protocols/mcp.md b/docs/protocols/mcp.md new file mode 100644 index 0000000..13da0f6 --- /dev/null +++ b/docs/protocols/mcp.md @@ -0,0 +1,266 @@ +--- +id: mcp +title: MCP Protocol +sidebar_position: 6 +--- + +# MCP Protocol Plugin + +The MCP (Model Context Protocol) plugin provides interoperability between UTCP and existing MCP servers, enabling gradual migration from MCP to UTCP while maintaining compatibility with existing MCP tools. + +## Call Template Structure + +```json +{ + "call_template_type": "mcp", + "server_config": { + "command": "node", + "args": ["mcp-server.js"], + "working_directory": "/app/mcp", + "env": { + "NODE_ENV": "production", + "LOG_LEVEL": "info" + }, + "timeout": 30 + }, + "connection_timeout": 10, + "request_timeout": 30 +} +``` + +## Server Configuration + +### Command-based Servers +```json +{ + "server_config": { + "command": "python", + "args": ["-m", "mcp_server", "--config", "config.json"], + "working_directory": "/app", + "env": { + "PYTHONPATH": "/app/lib", + "API_KEY": "${MCP_API_KEY}" + } + } +} +``` + +### HTTP-based Servers +```json +{ + "server_config": { + "transport": "http", + "url": "http://localhost:8080/mcp", + "headers": { + "Authorization": "Bearer ${MCP_TOKEN}" + } + } +} +``` + +## Migration Strategy + +The MCP protocol plugin enables a gradual migration path from MCP to native UTCP protocols: + +### Phase 1: MCP Integration +- Use existing MCP servers through UTCP +- No changes to MCP server code required +- UTCP client can call MCP tools seamlessly + +### Phase 2: Hybrid Approach +- Some tools use native UTCP protocols +- Legacy tools continue using MCP +- Gradual migration of high-value tools + +### Phase 3: Full Migration +- All tools use native UTCP protocols +- MCP servers deprecated +- Simplified architecture + +## Tool Discovery + +MCP servers expose tools through the standard MCP protocol. The UTCP MCP plugin: + +1. **Connects** to the MCP server using stdio or HTTP transport +2. **Discovers** available tools via MCP's `tools/list` method +3. **Maps** MCP tool definitions to UTCP tool format +4. **Registers** tools in the UTCP client + +## Request/Response Mapping + +### MCP to UTCP Tool Mapping +```json +// MCP Tool Definition +{ + "name": "read_file", + "description": "Read contents of a file", + "inputSchema": { + "type": "object", + "properties": { + "path": {"type": "string"} + }, + "required": ["path"] + } +} + +// UTCP Tool (after mapping) +{ + "name": "read_file", + "description": "Read contents of a file", + "inputs": { + "type": "object", + "properties": { + "path": {"type": "string"} + }, + "required": ["path"] + }, + "tool_call_template": { + "call_template_type": "mcp", + "server_config": {...} + } +} +``` + +### Request Flow +1. UTCP client receives tool call +2. MCP plugin formats request as MCP `tools/call` +3. Request sent to MCP server +4. MCP server processes and responds +5. Response mapped back to UTCP format + +## Authentication and Security + +### Server Authentication +```json +{ + "server_config": { + "command": "secure-mcp-server", + "env": { + "MCP_AUTH_TOKEN": "${MCP_SERVER_TOKEN}", + "MCP_CLIENT_ID": "${MCP_CLIENT_ID}" + } + } +} +``` + +### Transport Security +- **stdio**: Inherits process security model +- **HTTP**: Use HTTPS and proper authentication headers +- **WebSocket**: Use WSS and authentication tokens + +## Error Handling + +### Connection Errors +- Server startup failures +- Network connectivity issues +- Authentication failures +- Timeout errors + +### Protocol Errors +- Invalid MCP messages +- Unsupported MCP features +- Tool execution failures +- Resource access errors + +### Error Mapping +MCP errors are mapped to UTCP exceptions: +- `InvalidRequest` → `ValidationError` +- `MethodNotFound` → `ToolNotFoundError` +- `InternalError` → `ToolCallError` + +## Performance Considerations + +### Connection Management +- Persistent connections for stdio transport +- Connection pooling for HTTP transport +- Automatic reconnection on failures +- Graceful shutdown handling + +### Request Optimization +- Batch multiple tool calls when possible +- Cache tool discovery results +- Implement request timeouts +- Monitor response times + +## Limitations + +### MCP Feature Support +Not all MCP features are supported through UTCP: +- **Resources**: Not directly mapped to UTCP tools +- **Prompts**: Not supported in UTCP model +- **Sampling**: Not applicable to tool calling + +### Protocol Differences +- MCP's bidirectional communication vs UTCP's request/response +- MCP's resource model vs UTCP's tool-only model +- Different authentication mechanisms + +## Configuration Examples + +### Development Setup +```json +{ + "manual_call_templates": [{ + "name": "dev_mcp", + "call_template_type": "mcp", + "server_config": { + "command": "node", + "args": ["dev-server.js"], + "env": {"NODE_ENV": "development"} + }, + "connection_timeout": 5, + "request_timeout": 10 + }] +} +``` + +### Production Setup +```json +{ + "manual_call_templates": [{ + "name": "prod_mcp", + "call_template_type": "mcp", + "server_config": { + "transport": "http", + "url": "https://mcp.example.com/api", + "headers": { + "Authorization": "Bearer ${MCP_PROD_TOKEN}", + "X-Client-Version": "1.0.0" + } + }, + "connection_timeout": 30, + "request_timeout": 60 + }] +} +``` + +## Best Practices + +### Migration Planning +1. **Inventory** existing MCP servers and tools +2. **Prioritize** tools for migration based on usage +3. **Test** MCP integration thoroughly +4. **Monitor** performance and reliability +5. **Migrate** incrementally to native UTCP protocols + +### Monitoring and Debugging +- Enable debug logging for MCP communication +- Monitor server health and response times +- Track tool usage patterns +- Log authentication failures +- Set up alerts for connection issues + +### Security +- Use secure transport methods (HTTPS, WSS) +- Implement proper authentication +- Validate all inputs and outputs +- Monitor for suspicious activity +- Keep MCP servers updated + +## Language-Specific Implementation + +For implementation details and examples in your programming language: + +- **Multi-language**: [UTCP MCP Protocol Examples](https://github.com/universal-tool-calling-protocol) - MCP protocol examples across multiple languages +- **TypeScript**: [TypeScript MCP Protocol Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/blob/main/docs/protocols/mcp.md) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) diff --git a/docs/protocols/sse.md b/docs/protocols/sse.md new file mode 100644 index 0000000..93618e9 --- /dev/null +++ b/docs/protocols/sse.md @@ -0,0 +1,386 @@ +--- +id: sse +title: Server-Sent Events (SSE) +sidebar_position: 3 +--- + +# Server-Sent Events (SSE) + +The Server-Sent Events protocol plugin (`utcp-http`) enables UTCP to receive real-time streaming data from HTTP servers. SSE is perfect for tools that need to stream live updates, notifications, or continuous data feeds. + +## Installation + +SSE support is included with the HTTP plugin: + +```bash +# Example installation (Python) +pip install utcp-http + +# Example installation (Node.js) +npm install @utcp/http +``` + +## Call Template Structure + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "headers": { + "Authorization": "Bearer ${API_TOKEN}", + "Accept": "text/event-stream" + }, + "timeout": 60, + "max_events": 10, + "event_filter": { + "type": "data_update" + } +} +``` + +## Configuration Options + +The Server-Sent Events (SSE) call template enables real-time streaming data from HTTP endpoints. For complete field specifications and validation rules, see the [SSE Call Template API Reference](../api/plugins/communication_protocols/http/src/utcp_http/sse_call_template.md). +| `reconnect` | boolean | Auto-reconnect on connection loss (default: true) | +| `reconnect_delay` | number | Delay between reconnection attempts (default: 3) | + +## Authentication + +SSE uses standard HTTP authentication methods: + +### Bearer Token + +```json +{ + "headers": { + "Authorization": "Bearer ${ACCESS_TOKEN}" + } +} +``` + +### API Key + +```json +{ + "headers": { + "X-API-Key": "${API_KEY}" + } +} +``` + +### Query Parameter Auth + +```json +{ + "query_params": { + "token": "${API_TOKEN}", + "user_id": "${USER_ID}" + } +} +``` + +## Event Handling + +### Basic Event Stream + +```json +{ + "name": "stream_notifications", + "description": "Stream real-time notifications", + "inputs": { + "type": "object", + "properties": { + "user_id": {"type": "string"} + }, + "required": ["user_id"] + }, + "tool_call_template": { + "call_template_type": "sse", + "url": "https://api.example.com/notifications/stream", + "query_params": { + "user_id": "${user_id}" + }, + "headers": { + "Authorization": "Bearer ${ACCESS_TOKEN}" + }, + "timeout": 300 + } +} +``` + +### Filtered Events + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "event_filter": { + "type": "order_update", + "status": ["completed", "cancelled"] + }, + "max_events": 5 +} +``` + +## Examples + +### Stock Price Stream + +```json +{ + "name": "stream_stock_prices", + "description": "Stream real-time stock price updates", + "inputs": { + "type": "object", + "properties": { + "symbols": { + "type": "array", + "items": {"type": "string"} + }, + "duration": {"type": "number", "default": 60} + }, + "required": ["symbols"] + }, + "tool_call_template": { + "call_template_type": "sse", + "url": "https://api.stocks.com/stream", + "query_params": { + "symbols": "${symbols}", + "format": "json" + }, + "headers": { + "Authorization": "Bearer ${STOCK_API_KEY}" + }, + "timeout": "${duration}", + "event_filter": { + "type": "price_update" + } + } +} +``` + +### Log Monitoring + +```json +{ + "name": "monitor_logs", + "description": "Monitor application logs in real-time", + "inputs": { + "type": "object", + "properties": { + "service": {"type": "string"}, + "level": {"type": "string", "enum": ["error", "warn", "info", "debug"]} + }, + "required": ["service"] + }, + "tool_call_template": { + "call_template_type": "sse", + "url": "https://logs.example.com/stream", + "query_params": { + "service": "${service}", + "level": "${level}" + }, + "headers": { + "X-API-Key": "${LOG_API_KEY}" + }, + "timeout": 600, + "max_events": 100 + } +} +``` + +### System Metrics Stream + +```json +{ + "name": "stream_metrics", + "description": "Stream system performance metrics", + "inputs": { + "type": "object", + "properties": { + "metrics": { + "type": "array", + "items": {"type": "string"} + }, + "interval": {"type": "number", "default": 5} + }, + "required": ["metrics"] + }, + "tool_call_template": { + "call_template_type": "sse", + "url": "https://monitoring.example.com/metrics/stream", + "query_params": { + "metrics": "${metrics}", + "interval": "${interval}" + }, + "headers": { + "Authorization": "Bearer ${MONITORING_TOKEN}" + }, + "timeout": 300, + "reconnect": true, + "reconnect_delay": 5 + } +} +``` + +## Event Format + +SSE events follow the standard format: + +``` +event: message +data: {"type": "update", "value": 123} +id: event-123 +retry: 3000 + +event: heartbeat +data: {"timestamp": "2024-01-15T10:30:00Z"} + +data: {"message": "Simple data without event type"} +``` + +### Parsed Event Structure + +```json +{ + "event": "message", + "data": {"type": "update", "value": 123}, + "id": "event-123", + "retry": 3000, + "timestamp": "2024-01-15T10:30:00Z" +} +``` + +## Response Handling + +### Single Event Response + +```json +{ + "events": [ + { + "event": "data", + "data": {"result": "success"}, + "id": "1", + "timestamp": "2024-01-15T10:30:00Z" + } + ], + "total_events": 1, + "duration": 2.5 +} +``` + +### Multiple Events Response + +```json +{ + "events": [ + { + "event": "start", + "data": {"status": "processing"}, + "id": "1" + }, + { + "event": "progress", + "data": {"percent": 50}, + "id": "2" + }, + { + "event": "complete", + "data": {"result": "success"}, + "id": "3" + } + ], + "total_events": 3, + "duration": 15.2 +} +``` + +## Error Handling + +| Error Type | Description | Handling | +|------------|-------------|----------| +| Connection Failed | Cannot connect to SSE endpoint | Raise `SSEConnectionError` | +| Stream Timeout | No events received within timeout | Return partial results | +| Parse Error | Invalid SSE event format | Skip malformed events | +| Authentication Failed | Invalid credentials | Raise `SSEAuthError` | +| Server Error | HTTP 5xx response | Raise `SSEServerError` | + +## Best Practices + +1. **Set Appropriate Timeouts**: Configure timeouts based on expected data frequency +2. **Handle Reconnections**: Enable auto-reconnect for long-running streams +3. **Filter Events**: Use event filters to reduce unnecessary data processing +4. **Monitor Performance**: Track event rates and processing times +5. **Validate Data**: Validate incoming event data against expected schemas +6. **Handle Backpressure**: Implement buffering for high-frequency events +7. **Graceful Shutdown**: Properly close streams when done + +## Advanced Features + +### Custom Event Parsing + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "event_parser": { + "format": "json", + "extract_fields": ["timestamp", "level", "message"] + } +} +``` + +### Event Aggregation + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/metrics", + "aggregation": { + "window": 10, + "function": "average", + "field": "value" + } +} +``` + +### Conditional Termination + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "termination_condition": { + "event_type": "complete", + "data_field": "status", + "value": "finished" + } +} +``` + +## Common Use Cases + +- **Real-time Dashboards**: Live metrics, status updates +- **Notifications**: User alerts, system notifications +- **Log Streaming**: Application logs, audit trails +- **Progress Tracking**: Long-running task progress +- **Live Data Feeds**: News, social media, sensor data +- **Chat Applications**: Message streams, typing indicators + +## Protocol Comparison + +| Feature | SSE | WebSocket | HTTP Polling | +|---------|-----|-----------|--------------| +| Server-to-Client | ✅ | ✅ | ✅ | +| Client-to-Server | ❌ | ✅ | ✅ | +| Auto-Reconnect | ✅ | Manual | Manual | +| Overhead | Low | Low | High | +| Browser Support | ✅ | ✅ | ✅ | +| Simplicity | High | Medium | High | + +## Related Protocols + +- [HTTP](./http.md) - For request/response patterns +- [WebSocket](./websocket.md) - For bidirectional communication +- [Streamable HTTP](./streamable-http.md) - For chunked HTTP responses diff --git a/docs/protocols/streamable-http.md b/docs/protocols/streamable-http.md new file mode 100644 index 0000000..3ed9e6e --- /dev/null +++ b/docs/protocols/streamable-http.md @@ -0,0 +1,105 @@ +--- +id: streamable-http +title: Streamable HTTP Protocol +sidebar_position: 2 +--- + +# Streamable HTTP Protocol + +The Streamable HTTP protocol plugin (`utcp-http`) enables UTCP to handle large HTTP responses by streaming them in chunks. This is ideal for tools that return large datasets, files, or progressive results that don't fit into a single response payload. It leverages HTTP Chunked Transfer Encoding. + +## Call Template Structure + +```json +{ + "call_template_type": "streamable_http", + "url": "https://api.example.com/download/large-file", + "http_method": "GET", + "headers": { + "Accept": "application/octet-stream" + }, + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "Authorization", + "location": "header" + }, + "chunk_size": 8192, + "timeout": 300000, + "body_field": "request_data", + "header_fields": ["custom_header_arg"] +} +``` + +## Configuration Options + +The Streamable HTTP call template provides a way to configure streaming from HTTP endpoints. + +| Option | Type | Default | Description | +|---|---|---|---| +| `url` | string | **Required** | The streaming HTTP endpoint URL. Supports path parameters like `/users/{user_id}`. | +| `http_method` | string | `GET` | The HTTP method to use. Supported methods are `GET` and `POST`. | +| `content_type` | string | `application/octet-stream` | The `Content-Type` header to set for the request, especially when `body_field` is used. | +| `chunk_size` | integer | `4096` | The size of each data chunk in bytes to read from the stream. | +| `timeout` | integer | `60000` | Request timeout in milliseconds. | +| `headers` | object | `null` | Optional static headers to include in every request. | +| `auth` | object | `null` | Optional authentication configuration. See [HTTP Authentication](./http.md#authentication-methods). | +| `body_field` | string | `null` | The name of a single tool argument to be sent as the HTTP request body. | +| `header_fields` | array | `null` | A list of tool argument names to be sent as request headers. | + +## Response Handling + +The protocol processes the incoming stream based on the `Content-Type` header of the response: + +- **`application/x-ndjson`**: The stream is parsed as Newline Delimited JSON. Each line is yielded as a separate JSON object. +- **`application/octet-stream`**: The stream is yielded in binary chunks of the specified `chunk_size`. +- **`application/json`**: The entire response is buffered and yielded as a single JSON object. This is for endpoints that stream a single, large JSON document. +- **Other Types**: For any other `Content-Type`, the response is treated as a binary stream and yielded in chunks of `chunk_size`. + +## Authentication + +Streamable HTTP supports the same authentication methods as the standard HTTP protocol, including API Key, Basic Auth, and OAuth2. The configuration is identical. + +For more details, see the [HTTP Authentication Methods](./http.md#authentication-methods) documentation. + +## Variable Substitution + +Path parameters, query parameters, headers, and authentication fields all support variable substitution from tool arguments and environment variables, following the same syntax as the standard HTTP protocol. + +Example: +```json +{ + "url": "https://api.example.com/files/{file_id}/download", + "headers": { + "Authorization": "Bearer ${ACCESS_TOKEN}" + } +} +``` +Here, `{file_id}` is substituted from a tool argument, and `${ACCESS_TOKEN}` is substituted from an environment or configuration variable. + +For more details, see the [HTTP Variable Substitution](./http.md#variable-substitution) documentation. + +## Security Considerations + +- **SSL/TLS**: It is strongly recommended to use `https://` endpoints to protect data in transit. The implementation enforces HTTPS or localhost connections by default. +- **Authentication**: Never hardcode credentials. Use variable substitution to inject secrets from a secure source (e.g., environment variables). +- **Input Sanitization**: Ensure that any arguments used in URL path parameters or query strings are properly validated and sanitized to prevent injection attacks. + +## Error Handling + +Errors are handled similarly to the standard HTTP protocol: + +| Status Code | Error Type | Description | +|---|---|---| +| 400 | Bad Request | Invalid request parameters or body. | +| 401 | Unauthorized | Authentication failed or is required. | +| 403 | Forbidden | The authenticated user does not have permission. | +| 404 | Not Found | The requested resource or endpoint does not exist. | +| 5xx | Server Error | An error occurred on the server side. | + +Connection errors, timeouts, and other network issues will also be raised as exceptions. + +## Related Protocols + +- [HTTP](./http.md) - For standard request/response interactions. +- [Server-Sent Events (SSE)](./sse.md) - For unidirectional, real-time event streams from server to client. \ No newline at end of file diff --git a/docs/protocols/text.md b/docs/protocols/text.md new file mode 100644 index 0000000..dcca388 --- /dev/null +++ b/docs/protocols/text.md @@ -0,0 +1,437 @@ +--- +id: text +title: Text Protocol +sidebar_position: 5 +--- + +# Text Protocol + +The Text protocol plugin (`utcp-text`) enables UTCP to read and process text files from local filesystem or remote URLs. This is useful for tools that need to access documentation, configuration files, logs, or any text-based data. + +## Installation + +```bash +# Example installation (Python) +pip install utcp-text + +# Example installation (Node.js) +npm install @utcp/text +``` + +## Call Template Structure + +```json +{ + "call_template_type": "text", + "file_path": "/path/to/file.txt", + "encoding": "utf-8", + "max_size": 1048576, + "line_range": { + "start": 1, + "end": 100 + } +} +``` + +## Configuration Options + +The Text call template enables reading and processing text files from local filesystem or URLs. For complete field specifications and validation rules, see the [Text Call Template API Reference](../api/plugins/communication_protocols/text/src/utcp_text/text_call_template.md). + +## File Sources + +### Local Files + +```json +{ + "call_template_type": "text", + "file_path": "/var/log/application.log", + "encoding": "utf-8" +} +``` + +### Remote URLs + +```json +{ + "call_template_type": "text", + "file_path": "https://example.com/config.txt", + "max_size": 512000 +} +``` + +### Variable Substitution + +```json +{ + "call_template_type": "text", + "file_path": "/data/${filename}", + "encoding": "${file_encoding}" +} +``` + +## Examples + +### Read Configuration File + +```json +{ + "name": "read_config", + "description": "Read application configuration file", + "inputs": { + "type": "object", + "properties": { + "config_name": {"type": "string"} + }, + "required": ["config_name"] + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "/etc/app/${config_name}.conf", + "encoding": "utf-8", + "max_size": 65536 + } +} +``` + +### Read Log File with Line Range + +```json +{ + "name": "read_recent_logs", + "description": "Read recent log entries", + "inputs": { + "type": "object", + "properties": { + "log_file": {"type": "string"}, + "lines": {"type": "number", "default": 100} + }, + "required": ["log_file"] + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "/var/log/${log_file}", + "line_range": { + "start": -${lines}, + "end": -1 + } + } +} +``` + +### Read Remote Documentation + +```json +{ + "name": "fetch_documentation", + "description": "Fetch documentation from remote URL", + "inputs": { + "type": "object", + "properties": { + "doc_url": {"type": "string"}, + "section": {"type": "string"} + }, + "required": ["doc_url"] + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "${doc_url}", + "pattern": "(?s)## ${section}.*?(?=## |$)", + "max_size": 2097152 + } +} +``` + +### Search in File + +```json +{ + "name": "search_in_file", + "description": "Search for pattern in text file", + "inputs": { + "type": "object", + "properties": { + "file_path": {"type": "string"}, + "search_pattern": {"type": "string"} + }, + "required": ["file_path", "search_pattern"] + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "${file_path}", + "pattern": "${search_pattern}", + "transform": "strip" + } +} +``` + +## Line Range Options + +### Absolute Line Numbers + +```json +{ + "line_range": { + "start": 10, + "end": 50 + } +} +``` + +### Relative to End (Tail) + +```json +{ + "line_range": { + "start": -100, + "end": -1 + } +} +``` + +### From Start (Head) + +```json +{ + "line_range": { + "start": 1, + "end": 100 + } +} +``` + +## Pattern Matching + +### Simple Text Search + +```json +{ + "pattern": "ERROR" +} +``` + +### Regex Pattern + +```json +{ + "pattern": "\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2} ERROR.*" +} +``` + +### Multi-line Pattern + +```json +{ + "pattern": "(?s)START.*?END" +} +``` + +## Content Transformations + +### Case Transformations + +```json +{ + "transform": "upper" // Convert to uppercase +} +``` + +```json +{ + "transform": "lower" // Convert to lowercase +} +``` + +### Whitespace Handling + +```json +{ + "transform": "strip" // Remove leading/trailing whitespace +} +``` + +### Custom Transformations + +```json +{ + "transform": "normalize_whitespace" // Normalize all whitespace +} +``` + +## Response Format + +### Successful Read + +```json +{ + "content": "File content here...", + "metadata": { + "file_path": "/path/to/file.txt", + "size": 1024, + "lines": 25, + "encoding": "utf-8", + "last_modified": "2024-01-15T10:30:00Z" + } +} +``` + +### Filtered Content + +```json +{ + "content": "Matching lines...", + "metadata": { + "file_path": "/path/to/file.txt", + "total_lines": 1000, + "matched_lines": 5, + "pattern": "ERROR", + "line_range": {"start": 1, "end": 100} + } +} +``` + +## Error Handling + +| Error Type | Description | Handling | +|------------|-------------|----------| +| File Not Found | File doesn't exist | Raise `FileNotFoundError` | +| Permission Denied | No read permission | Raise `PermissionError` | +| File Too Large | Exceeds max_size limit | Raise `FileSizeError` | +| Encoding Error | Invalid file encoding | Raise `EncodingError` | +| Network Error | URL fetch failed | Raise `NetworkError` | + +## Security Considerations + +### Path Traversal Prevention + +```json +{ + "call_template_type": "text", + "file_path": "/safe/directory/${filename}", + "allowed_paths": ["/safe/directory/"] +} +``` + +### File Size Limits + +```json +{ + "max_size": 1048576 // 1MB limit +} +``` + +### URL Restrictions + +```json +{ + "allowed_domains": ["example.com", "docs.company.com"] +} +``` + +## Best Practices + +1. **Set Size Limits**: Always set appropriate max_size limits +2. **Validate Paths**: Validate file paths to prevent directory traversal +3. **Handle Encoding**: Specify encoding explicitly for non-UTF-8 files +4. **Use Line Ranges**: Use line ranges for large files to improve performance +5. **Pattern Efficiency**: Use efficient regex patterns for content filtering +6. **Cache Results**: Cache frequently accessed files +7. **Monitor Access**: Log file access for security auditing + +## Advanced Features + +### Conditional Reading + +```json +{ + "call_template_type": "text", + "file_path": "/var/log/app.log", + "condition": { + "modified_since": "2024-01-15T00:00:00Z" + } +} +``` + +### Multi-file Reading + +```json +{ + "call_template_type": "text", + "file_paths": [ + "/etc/app/config1.txt", + "/etc/app/config2.txt" + ], + "merge_strategy": "concatenate" +} +``` + +### Streaming Large Files + +```json +{ + "call_template_type": "text", + "file_path": "/var/log/huge.log", + "streaming": true, + "chunk_size": 8192 +} +``` + +## Common Use Cases + +- **Configuration Management**: Reading config files, environment files +- **Log Analysis**: Processing application logs, system logs +- **Documentation**: Accessing README files, API docs, manuals +- **Data Processing**: Reading CSV, JSON, XML text files +- **Template Processing**: Reading template files for generation +- **Code Analysis**: Reading source code files for analysis +- **Monitoring**: Reading status files, health check files + +## Performance Considerations + +| File Size | Recommended Approach | +|-----------|---------------------| +| < 1MB | Read entire file | +| 1MB - 10MB | Use line ranges | +| 10MB - 100MB | Use streaming | +| > 100MB | Use external tools | + +## Integration Examples + +### With HTTP Protocol + +```json +{ + "name": "process_uploaded_file", + "description": "Process uploaded text file", + "inputs": { + "type": "object", + "properties": { + "file_url": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "${file_url}", + "max_size": 5242880 + } +} +``` + +### With CLI Protocol + +```json +{ + "name": "analyze_log_file", + "description": "Analyze log file with external tool", + "inputs": { + "type": "object", + "properties": { + "log_path": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "cli", + "command": "log-analyzer", + "args": ["--file", "${log_path}", "--format", "json"] + } +} +``` diff --git a/docs/protocols/websocket.md b/docs/protocols/websocket.md new file mode 100644 index 0000000..5ce7789 --- /dev/null +++ b/docs/protocols/websocket.md @@ -0,0 +1,355 @@ +--- +id: websocket +title: WebSocket Protocol +sidebar_position: 2 +--- + +# WebSocket Protocol + +The WebSocket protocol plugin (`utcp-websocket`) enables UTCP to communicate with WebSocket servers for real-time, bidirectional communication. This is ideal for tools that require persistent connections or real-time updates. + +## Installation + +```bash +# Example installation (Python) +pip install utcp-websocket + +# Example installation (Node.js) +npm install @utcp/websocket +``` + +## Call Template Structure + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": { + "type": "request", + "action": "${action}", + "data": "${data}" + }, + "connection_timeout": 10, + "response_timeout": 30, + "auth": { + "auth_type": "api_key", + "api_key": "${WS_API_KEY}", + "location": "query" + } +} +``` + +## Configuration Options + +The WebSocket call template enables real-time communication with WebSocket servers. For complete field specifications and validation rules, see the WebSocket Call Template API Reference (WIP). +| `expected_responses` | number | Number of expected response messages (default: 1) | +| `ping_interval` | number | Ping interval in seconds (default: 30) | + +## Authentication + +WebSocket authentication can be handled in several ways: + +### Query Parameter Authentication + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "token", + "location": "query" + } +} +``` + +### Header Authentication + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +### Message-based Authentication + +```json +{ + "message": { + "type": "auth", + "token": "${API_KEY}" + } +} +``` + +## Message Formats + +### JSON Messages + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": { + "id": "{{uuid}}", + "method": "getData", + "params": { + "query": "${query}", + "limit": 10 + } + } +} +``` + +### Text Messages + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": "GET_DATA:${query}" +} +``` + +### Binary Messages + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": { + "type": "binary", + "data": "${base64_data}" + } +} +``` + +## Examples + +### Real-time Data Subscription + +```json +{ + "name": "subscribe_stock_price", + "description": "Subscribe to real-time stock price updates", + "inputs": { + "type": "object", + "properties": { + "symbol": {"type": "string"}, + "duration": {"type": "number", "default": 60} + }, + "required": ["symbol"] + }, + "tool_call_template": { + "call_template_type": "websocket", + "url": "wss://api.stocks.com/ws", + "message": { + "action": "subscribe", + "symbol": "${symbol}", + "type": "price" + }, + "response_timeout": "${duration}", + "expected_responses": -1, + "close_after_response": false + } +} +``` + +### Chat Bot Integration + +```json +{ + "name": "send_chat_message", + "description": "Send a message to the chat bot", + "inputs": { + "type": "object", + "properties": { + "message": {"type": "string"}, + "user_id": {"type": "string"} + }, + "required": ["message", "user_id"] + }, + "tool_call_template": { + "call_template_type": "websocket", + "url": "wss://chat.example.com/ws", + "message": { + "type": "message", + "user_id": "${user_id}", + "content": "${message}", + "timestamp": "{{now}}" + }, + "headers": { + "Authorization": "Bearer ${CHAT_TOKEN}" + } + } +} +``` + +### IoT Device Control + +```json +{ + "name": "control_device", + "description": "Send control commands to IoT device", + "inputs": { + "type": "object", + "properties": { + "device_id": {"type": "string"}, + "command": {"type": "string"}, + "value": {"type": "number"} + }, + "required": ["device_id", "command"] + }, + "tool_call_template": { + "call_template_type": "websocket", + "url": "wss://iot.example.com/device/${device_id}", + "message": { + "command": "${command}", + "value": "${value}", + "timestamp": "{{now}}" + }, + "connection_timeout": 5, + "response_timeout": 10 + } +} +``` + +## Connection Management + +### Persistent Connections + +For tools that need to maintain persistent connections: + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": {"action": "ping"}, + "close_after_response": false, + "ping_interval": 30 +} +``` + +### Connection Pooling + +The WebSocket protocol automatically manages connection pooling for efficiency: + +- Reuses connections to the same endpoint +- Handles connection lifecycle automatically +- Implements reconnection logic for dropped connections + +## Response Handling + +### Single Response + +```json +{ + "expected_responses": 1, + "close_after_response": true +} +``` + +### Multiple Responses + +```json +{ + "expected_responses": 5, + "response_timeout": 60 +} +``` + +### Streaming Responses + +```json +{ + "expected_responses": -1, + "response_timeout": 300, + "close_after_response": false +} +``` + +## Error Handling + +| Error Type | Description | Handling | +|------------|-------------|----------| +| Connection Failed | Cannot establish WebSocket connection | Raise `WebSocketConnectionError` | +| Authentication Failed | WebSocket handshake authentication failed | Raise `WebSocketAuthError` | +| Timeout | No response within timeout period | Raise `WebSocketTimeoutError` | +| Protocol Error | Invalid WebSocket protocol usage | Raise `WebSocketProtocolError` | +| Connection Closed | Server closed connection unexpectedly | Raise `WebSocketClosedError` | + +## Best Practices + +1. **Use Secure WebSockets**: Always use `wss://` for production +2. **Handle Reconnections**: Implement retry logic for connection failures +3. **Set Appropriate Timeouts**: Configure timeouts based on expected response times +4. **Validate Messages**: Validate both outgoing and incoming messages +5. **Monitor Connections**: Track connection health and performance +6. **Implement Heartbeats**: Use ping/pong for connection health checks +7. **Handle Backpressure**: Manage message queuing for high-throughput scenarios + +## Advanced Features + +### Message Filtering + +Filter incoming messages based on criteria: + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": {"subscribe": "all"}, + "message_filter": { + "type": "stock_price", + "symbol": "${symbol}" + } +} +``` + +### Custom Headers + +Include custom headers in the WebSocket handshake: + +```json +{ + "headers": { + "User-Agent": "UTCP-Client/1.0", + "X-Client-ID": "${CLIENT_ID}", + "Authorization": "Bearer ${TOKEN}" + } +} +``` + +### Compression + +Enable WebSocket compression: + +```json +{ + "compression": "deflate", + "compression_threshold": 1024 +} +``` + +## Common Use Cases + +- **Real-time Data**: Stock prices, sensor data, live metrics +- **Chat Applications**: Messaging, notifications, presence +- **Gaming**: Real-time game state, multiplayer coordination +- **IoT Control**: Device commands, status updates +- **Live Updates**: News feeds, social media streams +- **Collaborative Tools**: Document editing, shared whiteboards + +## Protocol Comparison + +| Feature | WebSocket | HTTP | SSE | +|---------|-----------|------|-----| +| Bidirectional | ✅ | ❌ | ❌ | +| Real-time | ✅ | ❌ | ✅ | +| Persistent | ✅ | ❌ | ✅ | +| Overhead | Low | High | Medium | +| Complexity | Medium | Low | Low | diff --git a/docs/security.md b/docs/security.md index b182938..fbf10f6 100644 --- a/docs/security.md +++ b/docs/security.md @@ -6,123 +6,439 @@ sidebar_position: 6 # Security Considerations -Security is a critical aspect of any protocol that enables tool access and execution. This section outlines key security considerations when implementing and using UTCP. +Security is critical when enabling direct tool access through UTCP. This guide covers security considerations specific to UTCP's "manual" approach and provides practical guidance for secure implementations. -## Authentication +## UTCP Security Model -UTCP supports several authentication methods across different communication protocol types: +### Direct Communication Implications -### API Key Authentication +UTCP's direct communication model has unique security characteristics: + +**Advantages:** +- No middleman to compromise +- Native security controls remain intact +- Reduced attack surface (no proxy servers) +- Existing monitoring and logging continue to work + +**Considerations:** +- Clients must handle multiple authentication methods +- Manual endpoints become discovery targets +- Variable substitution introduces injection risks + +### Manual Discovery Security + +Secure your UTCP manual endpoints: + +**Best Practices:** +- Implement authentication for manual discovery endpoints +- Validate discovery access tokens +- Return only tools the client is authorized to see +- Filter tools based on user permissions +- Log all manual discovery attempts + +## Authentication & Authorization + +### Enhanced Authentication Examples + +#### API Key with Rotation ```json { "auth": { "auth_type": "api_key", - "api_key": "YOUR_API_KEY", + "api_key": "${API_KEY}", "var_name": "X-API-Key", "location": "header" } } ``` -The `location` field specifies where the API key is placed, and can be `header`, `query`, or `cookie`. +**Secure implementation:** +- Use rotating API keys with current and next key support +- Implement automatic key rotation based on time +- Store keys securely in environment variables +- Validate keys before processing requests -### Basic Authentication +#### OAuth2 with Scope Validation ```json { "auth": { - "auth_type": "basic", - "username": "user", - "password": "pass" + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "tools:read tools:execute" } } ``` -### OAuth2 Authentication +#### Per-Tool Authorization ```json { - "auth": { - "auth_type": "oauth2", - "client_id": "YOUR_CLIENT_ID", - "client_secret": "YOUR_CLIENT_SECRET", - "token_url": "https://auth.example.com/token", - "scope": "read:tools" + "name": "admin_tool", + "description": "Administrative operations", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/admin/action", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "${ADMIN_TOKEN}", + "var_name": "Authorization", + "location": "header" + } } } ``` -The `scope` field is optional and specifies the level of access that the client is requesting. +## Protocol-Specific Security -## Tool Access Control +### HTTP/HTTPS Security -When exposing tools through UTCP, consider implementing these access controls: - -1. **Tool-Level Permissions**: Define which users/agents can access specific tools -2. **Parameter Constraints**: Restrict parameter values to prevent abuse -3. **Rate Limiting**: Implement per-user/per-tool rate limits -4. **Usage Quotas**: Set maximum usage quotas for tools -5. **Audit Logging**: Log all tool calls for security monitoring +**Required configurations:** +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "verify_ssl": true, + "timeout": 30, + "headers": { + "User-Agent": "UTCP-Client/1.0", + "X-Request-ID": "${request_id}" + } +} +``` -## Communication Protocol-Specific Considerations +**Security checklist:** +- ✅ Always use HTTPS in production +- ✅ Validate SSL certificates (`verify_ssl: true`) +- ✅ Set appropriate timeouts +- ✅ Include request tracking headers +- ✅ Implement retry limits -### HTTP +### WebSocket Security -- Always use HTTPS, never HTTP +**Secure WebSocket configuration:** +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "headers": { + "Authorization": "Bearer ${WS_TOKEN}", + "Origin": "https://trusted-domain.com" + }, + "ping_interval": 30, + "connection_timeout": 10 +} +``` -### CLI +**Security measures:** +- ✅ Use WSS (secure WebSocket) only +- ✅ Validate Origin headers +- ✅ Implement connection timeouts +- ✅ Use heartbeat/ping for connection health +- ✅ Limit concurrent connections per client -:::important +### CLI Security -CLI poses a significant security risk as it executes commands on the local system. +:::danger High Risk Protocol +CLI execution poses significant security risks. Use with extreme caution. ::: -- Validate and sanitize all input parameters -- Run commands with the minimum necessary permissions -- Implement allow-lists for permitted commands -- Sandbox execution environments when possible +**Secure CLI implementation:** +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "cd /safe/sandbox", + "append_to_final_output": false + }, + { + "command": "/usr/local/bin/safe-script --input UTCP_ARG_sanitized_input_UTCP_END", + "append_to_final_output": true + } + ], + "working_dir": "/safe/sandbox", + "env_vars": { + "PATH": "/usr/local/bin:/usr/bin", + "HOME": "/tmp/sandbox" + } +} +``` + +**Security requirements:** +- ✅ Use absolute paths for commands +- ✅ Sanitize all input parameters +- ✅ Run in sandboxed environments +- ✅ Limit environment variables +- ✅ Set strict timeouts +- ✅ Validate exit codes +- ✅ Use minimal user permissions + +**Input sanitization requirements:** +- Remove dangerous shell metacharacters: `;`, `&`, `|`, `` ` ``, `$`, `()`, `{}`, `[]`, `<>` +- Escape inputs appropriately for shell execution +- Use parameterized command execution when possible +- Validate inputs against expected patterns +- Implement length limits for all inputs + +### Server-Sent Events (SSE) Security + +**Secure SSE configuration:** +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "headers": { + "Authorization": "Bearer ${SSE_TOKEN}", + "Accept": "text/event-stream", + "Cache-Control": "no-cache" + }, + "timeout": 300, + "max_events": 1000 +} +``` + +**Security considerations:** +- ✅ Authenticate SSE connections +- ✅ Set maximum event limits +- ✅ Implement connection timeouts +- ✅ Validate event data format +- ✅ Monitor for event flooding + +### Text Protocol Security + +**Secure file access:** +```json +{ + "call_template_type": "text", + "file_path": "/safe/data/${filename}" +} +``` + +**Security measures:** +- ✅ Restrict file paths to safe directories +- ✅ Set maximum file size limits +- ✅ Validate file extensions +- ✅ Prevent directory traversal attacks +- ✅ Use safe encoding handling + +**Path validation requirements:** +- Resolve all paths to absolute paths +- Check if paths are within allowed directories +- Handle symbolic links by resolving them first +- Validate against directory traversal attacks (`../`) +- Return false for any path resolution errors +- Use allowlists of permitted directories + +### MCP Security + +**Secure MCP server configuration:** +```json +{ + "call_template_type": "mcp", + "config": { + "mcpServers": { + "server_name": { + "transport": "stdio", + "command": ["python", "-m", "my_mcp_server"] + } + } + } +} +``` + +**Security considerations:** +- ✅ Use trusted MCP server implementations +- ✅ Sandbox MCP server processes +- ✅ Limit server resource usage +- ✅ Monitor server health and logs +- ✅ Implement server restart policies + +## Input Validation & Sanitization + +### JSON Schema Validation + +**Comprehensive input validation:** +```json +{ + "inputs": { + "type": "object", + "properties": { + "email": { + "type": "string", + "format": "email", + "maxLength": 254 + }, + "age": { + "type": "integer", + "minimum": 0, + "maximum": 150 + }, + "tags": { + "type": "array", + "items": {"type": "string", "pattern": "^[a-zA-Z0-9_-]+$"}, + "maxItems": 10 + } + }, + "required": ["email"], + "additionalProperties": false + } +} +``` + +### Parameter Sanitization + +**Server-side validation requirements:** +- Use regex patterns to validate input formats (e.g., `^[a-zA-Z0-9_-]+$` for user IDs) +- Implement maximum length limits for all string inputs +- Remove dangerous characters like `<>`, `"`, `'` from user inputs +- Validate all inputs against expected schemas +- Sanitize inputs before processing + +## Secure Variable Handling + +### Environment Variable Security + +**Secure variable loading requirements:** +- Only allow variables with approved prefixes (e.g., `UTCP_`, `API_`) +- Validate variable names against allowlists +- Implement length limits for variable values (e.g., max 10,000 characters) +- Check for dangerous characters in values (`<`, `>`, `"`, `'`) +- Use secure variable substitution methods +- Log all variable access attempts + +### Runtime Variable Substitution + +**Secure substitution requirements:** + +Implement variable substitution with these security measures: +- Only substitute variables matching the pattern `${variable_name}` +- Validate variable names contain only alphanumeric characters and underscores +- Check that all variables exist before substitution +- Sanitize variable values to prevent injection attacks +- Reject values containing dangerous characters like `<`, `>`, `"`, `'`, `;`, `&` +- Limit variable value length to prevent buffer overflow attacks + +## Network & Transport Security + +### TLS Configuration + +**Minimum TLS requirements:** +**TLS configuration requirements:** + +Configure secure HTTP clients with these settings: +- Enable SSL certificate verification +- Set reasonable connection timeouts (e.g., 30 seconds) +- Limit maximum connections to prevent resource exhaustion +- Use TLS 1.2 or higher as minimum version +- Enable hostname verification +- Require certificate verification (CERT_REQUIRED mode) + +### Certificate Validation + +**Enhanced certificate validation:** + +Implement robust certificate validation: +- Use trusted certificate authority bundles +- Enable hostname verification against certificate +- Require valid certificate chains +- Set minimum TLS version to 1.2 +- Configure strong cipher suites (ECDHE+AESGCM, CHACHA20, DHE+AESGCM) +- Reject weak algorithms (aNULL, MD5, DSS) + +## Monitoring & Incident Response + +### Security Logging + +**Comprehensive security logging:** + +Implement security logging with these components: +- **Tool Call Logging**: Record all tool invocations with user ID, tool name, timestamp, success status, and parameters +- **Authentication Logging**: Log authentication attempts, failures, and reasons +- **Structured Format**: Use JSON format for easy parsing and analysis +- **Sensitive Data Protection**: Avoid logging sensitive information like passwords or tokens +- **Audit Trail**: Maintain immutable logs for compliance and forensic analysis + +### Anomaly Detection + +**Basic anomaly detection:** -### WebSocket +Implement anomaly detection with these features: +- **Rate Limiting**: Track requests per user with configurable limits (e.g., 60 calls/minute, 1000 calls/hour) +- **Time Window Management**: Clean old entries and maintain sliding time windows +- **Multi-tier Limits**: Enforce both short-term (per minute) and long-term (per hour) rate limits +- **Automatic Blocking**: Reject requests that exceed configured thresholds +- **Call Tracking**: Record timestamps of all user requests for analysis -- Use secure WebSocket (WSS) connections +## Security Testing & Validation -## Data Protection +### Testing Methodologies -1. **Data in Transit**: Ensure all communications use TLS 1.2+ encryption -2. **Data at Rest**: Encrypt sensitive configuration data -3. **Sensitive Data in Logs**: Prevent logging of sensitive parameters -4. **PII Handling**: Implement proper controls for personal information +**Security test examples:** -## Secure Implementation Checklist +Implement comprehensive security testing: +- **Injection Prevention Tests**: Test for SQL injection, command injection, and other malicious inputs +- **Path Traversal Tests**: Verify protection against directory traversal attacks (../../../etc/passwd) +- **Rate Limiting Tests**: Confirm rate limiting enforcement under high load +- **Authentication Tests**: Validate proper authentication and authorization +- **Input Validation Tests**: Test boundary conditions and malformed inputs -- [ ] Use HTTPS/WSS for all network communications -- [ ] Implement proper authentication for all communication protocols -- [ ] Validate all input against schemas before processing -- [ ] Sanitize inputs to prevent injection attacks -- [ ] Implement rate limiting to prevent abuse -- [ ] Set appropriate timeouts for all operations -- [ ] Log security-relevant events -- [ ] Regularly update dependencies -- [ ] Implement proper error handling that doesn't leak sensitive information +### Security Automation -## Common Vulnerabilities to Avoid +**Automated security checks:** + +Implement automated security validation: +- **Protocol Security**: Verify HTTPS usage instead of HTTP for web requests +- **Credential Detection**: Check for hardcoded passwords, secrets, or API keys +- **Variable Validation**: Ensure proper variable substitution patterns ($\{variable\}) +- **CLI Security**: Validate command-line tools use absolute paths and safe commands +- **URL Validation**: Check for suspicious or malformed URLs +- **Configuration Review**: Automated scanning of UTCP manuals for security issues -| Vulnerability | Prevention | -|--------------|------------| -| Injection Attacks | Validate and sanitize all inputs | -| Credential Leakage | Use secure credential storage | -| Excessive Permissions | Follow the principle of least privilege | -| Man-in-the-Middle | Use certificate validation and pinning | -| Denial of Service | Implement rate limiting and timeouts | -| Information Disclosure | Ensure errors don't leak sensitive data | +## Security Checklist -## Secure Development Lifecycle +### Tool Provider Security + +- [ ] UTCP manual endpoint requires authentication +- [ ] All tool endpoints use HTTPS/WSS +- [ ] Input validation implemented for all tools +- [ ] Rate limiting configured per user/tool +- [ ] Security logging enabled +- [ ] Credentials stored securely (not hardcoded) +- [ ] SSL certificate validation enabled +- [ ] Appropriate timeouts configured +- [ ] Error messages don't leak sensitive information -1. **Design**: Conduct threat modeling during protocol design -2. **Implementation**: Follow secure coding practices -3. **Testing**: Perform security testing and code reviews -4. **Deployment**: Use secure deployment practices -5. **Maintenance**: Monitor for security issues and update regularly +### Tool Consumer Security + +- [ ] Variable substitution is sanitized +- [ ] SSL certificate verification enabled +- [ ] Connection timeouts configured +- [ ] Rate limiting respected +- [ ] Security events logged +- [ ] Credentials rotated regularly +- [ ] Network connections monitored +- [ ] Input validation before tool calls -By following these security considerations, UTCP implementations can minimize risks while enabling powerful tool integrations across various communication protocols. +### Protocol-Specific Security + +- [ ] **HTTP**: HTTPS only, certificate validation +- [ ] **WebSocket**: WSS only, origin validation +- [ ] **CLI**: Sandboxed execution, input sanitization +- [ ] **SSE**: Authenticated connections, event limits +- [ ] **Text**: Path validation, size limits +- [ ] **MCP**: Trusted servers, resource limits + +By following these security guidelines, you can safely implement UTCP while maintaining strong security posture across all communication protocols. + +For protocol-specific security details, see: +- [HTTP Security](./protocols/http.md#security-considerations) +- [WebSocket Security](./protocols/websocket.md#security-considerations) +- [CLI Security](./protocols/cli.md#security-considerations) +- [SSE Security](./protocols/sse.md#security-considerations) +- [Text Security](./protocols/text.md#security-considerations) +- [MCP Security](./protocols/mcp.md#security-considerations) diff --git a/docs/utcp-vs-mcp.md b/docs/utcp-vs-mcp.md index 3dec676..bc670ff 100644 --- a/docs/utcp-vs-mcp.md +++ b/docs/utcp-vs-mcp.md @@ -4,99 +4,478 @@ title: UTCP vs MCP sidebar_position: 5 --- -# UTCP vs MCP tool calling: A Comparison +# UTCP vs MCP: A Comprehensive Comparison -This page compares the Universal Tool Calling Protocol (UTCP) with the Model Context Protocol's (MCP) tool calling functionality, highlighting their different approaches to agent-tool integration. +:::info Language Examples +This comparison uses **Python** examples. Both UTCP and MCP have implementations in multiple languages - check respective GitHub organizations for language-specific examples. +::: + +This guide compares the Universal Tool Calling Protocol (UTCP) with the Model Context Protocol (MCP), helping you choose the right approach for your AI tool integration needs. ## Video Overview -## Architectural Differences - -| Aspect | MCP | UTCP | -|--------|-----|------| -| **Core Model** | Middleman | Manual | -| **Architecture** | Agents ↔ MCP Server ↔ Tool | Agent ↔ Tool (Direct) | -| **Integration Approach** | Wraps existing tools | Describes how to call existing tools | -| **Network Hops** | Double (Agent → MCP → Tool) | Single (Agent → Tool) | -| **Protocol Dependency** | Hard dependency on protocol for every call | Protocol only needed during discovery | +## Quick Comparison -## The Middleman vs Manual Philosophies +| Aspect | UTCP | MCP | +|--------|------|-----| +| **Philosophy** | Manual (describes how to call tools) | Middleman (wraps tools in protocol) | +| **Architecture** | Agent → Tool (Direct) | Agent → MCP Server → Tool | +| **Infrastructure** | None required | Wrapper servers needed | +| **Protocols** | HTTP, WebSocket, CLI, SSE, etc. | JSON-RPC over stdio/HTTP | +| **Performance** | Native tool performance | Additional proxy overhead | +| **Maintenance** | Minimal | High (server maintenance) | -### The MCP "Middleman" Approach +## Architectural Differences -MCP positions itself as the "USB-C for AI Agents" — a universal adapter that all tools must plug into. This approach: +### UTCP: The "Manual" Approach + +UTCP provides a standardized way to describe how to call existing tools directly: + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [{ + "name": "get_weather", + "description": "Get current weather", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.weather.com/current", + "http_method": "GET", + "query_params": {"location": "${location}"} + } + }] +} +``` + +**Flow:** Agent discovers manual → Agent calls tool directly + +### MCP: The "Middleman" Approach + +MCP requires building servers that wrap your tools: + +**MCP Server Implementation Requirements:** +- Create a dedicated server process for each tool provider +- Implement tool listing functionality to expose available tools +- Implement tool calling handlers that proxy requests to actual APIs +- Maintain server infrastructure and handle client connections +- Route all tool calls through the MCP server layer + +**Flow:** Agent → MCP Server → Tool → MCP Server → Agent + +## Technical Comparison + +### Performance Impact + +#### UTCP Performance + +**Direct API calls with no overhead:** +- Make HTTP requests directly to weather service endpoints +- No intermediate proxy servers or additional network hops +- Latency equals API response time only (~100ms) +- Native HTTP client performance with connection pooling + +#### MCP Performance + +**Requires MCP server proxy:** +- Connect to MCP server before making tool calls +- Route requests through MCP server to actual weather API +- Additional network hop adds latency overhead +- Latency includes API response time plus MCP server processing (~150ms) + +### Infrastructure Requirements + +#### UTCP Infrastructure + +**Minimal infrastructure requirements:** +- Add single discovery endpoint to existing API (e.g., GET /utcp) +- Return static JSON manual describing available tools +- No additional servers, processes, or infrastructure needed +- Total infrastructure: 0 additional servers + +#### MCP Infrastructure + +**MCP infrastructure requirements:** +- Requires dedicated MCP server processes for each tool provider +- Process management, monitoring, and scaling infrastructure needed +- Client connection management and session handling required +- Total infrastructure: N MCP servers (one per tool provider) + +### Protocol Support Comparison + +#### UTCP Protocol Flexibility +```json +{ + "tools": [ + { + "name": "http_tool", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/data" + } + }, + { + "name": "websocket_tool", + "tool_call_template": { + "call_template_type": "websocket", + "url": "wss://api.example.com/ws" + } + }, + { + "name": "cli_tool", + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "git status --porcelain", + "append_to_final_output": true + } + ] + } + } + ] +} +``` + +#### MCP Protocol Limitation + +**MCP protocol constraints:** +- Only supports JSON-RPC over stdio/HTTP transport +- All tools must be wrapped in MCP server implementations +- Cannot directly call WebSocket, CLI, or other native protocols +- Requires protocol translation layer for non-HTTP tools -- Forces all traffic through a new protocol layer -- Requires writing "wrappers" for existing tools -- Needs to reinvent solutions for auth, security, and other infrastructure concerns +## Feature Comparison -This creates what we call the "wrapper tax": the additional infrastructure, development, and maintenance overhead required to adapt existing tools to work with MCP. +### Authentication & Security + +#### UTCP: Native Authentication +```json +{ + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/data", + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token" + } + } +} +``` + +**Benefits:** +- Uses existing authentication systems +- No credential translation needed +- Native rate limiting and monitoring +- Existing security policies apply + +#### MCP: Server-Mediated Authentication + +**MCP server authentication requirements:** +- MCP server must handle authentication translation between client and API +- Server stores and manages API credentials on behalf of clients +- Server makes authenticated calls to actual APIs using stored credentials +- Requires credential management and secure storage in MCP server + +**Challenges:** +- Credential management in MCP servers +- Additional security layer to maintain +- Auth translation complexity + +### Streaming & Real-time Data + +#### UTCP: Native Streaming Support +```json +{ + "name": "stream_logs", + "tool_call_template": { + "call_template_type": "sse", + "url": "https://api.example.com/logs/stream", + "timeout": 300 + } +} +``` + +#### MCP: Limited Streaming + +**MCP streaming limitations:** +- MCP has basic streaming capabilities but requires server implementation +- More complex to set up and maintain than native streaming protocols +- Additional abstraction layer between client and streaming data source + +### Error Handling + +#### UTCP: Native Error Responses + +**Direct error handling:** +- Errors come directly from the tool without translation +- Native HTTP status codes and error messages preserved +- Full error context available including headers and response body +- No error translation or abstraction layer + +#### MCP: Wrapped Error Responses + +**Error abstraction layer:** +- Errors are wrapped and translated by MCP server +- Original error context may be lost in translation +- MCP-specific error format instead of native tool errors +- Additional debugging complexity due to error wrapping + +## Migration & Interoperability + +### Migrating from MCP to UTCP + +UTCP provides an MCP plugin for gradual migration: + +**Migration Strategy:** +- **Phase 1**: Use existing MCP servers via UTCP's MCP protocol plugin +- Configure UTCP client to connect to legacy MCP servers using MCP call templates +- **Phase 2**: Migrate high-value tools to native UTCP protocols (HTTP, WebSocket, CLI) +- **Phase 3**: Deprecate MCP servers once migration is complete + +[**Complete migration guide →**](./protocols/mcp.md) + +### Hybrid Approach + +You can use both protocols simultaneously: + +**Hybrid approach during migration:** +- Configure UTCP client with both native UTCP and legacy MCP call templates +- Native UTCP tools use direct HTTP/WebSocket/CLI protocols +- Legacy MCP tools continue using MCP protocol plugin +- Gradually migrate tools from MCP to native UTCP protocols +- Single client interface for both native and legacy tools + +## Enterprise Decision Factors + +### Total Cost of Ownership + +#### UTCP TCO +``` +Infrastructure: $0 (uses existing APIs) +Development: Low (add one endpoint) +Maintenance: Minimal (static JSON) +Scaling: Automatic (scales with existing API) +Monitoring: Existing tools work +``` + +#### MCP TCO +``` +Infrastructure: High (dedicated servers) +Development: High (build wrapper servers) +Maintenance: High (server management) +Scaling: Complex (scale MCP servers separately) +Monitoring: Additional monitoring stack needed +``` -### The UTCP "Manual" Approach +### Development Velocity -UTCP takes a different approach — it's a "manual" that describes how to call tools directly: +#### UTCP Development Speed + +**Rapid deployment timeline:** +- **Day 1**: Add UTCP discovery endpoint to existing API +- **Day 2**: Tools are immediately available to AI agents +- No additional infrastructure, servers, or deployment needed +- Minimal code changes to existing systems -- Provides all necessary information to call native APIs directly -- Gets out of the way after tool discovery -- Leverages existing infrastructure for auth, security, etc. +#### MCP Development Speed -This eliminates the wrapper tax and allows organizations to expose their existing APIs to AI agents without building and maintaining additional infrastructure. +**Extended development timeline:** +- **Week 1-2**: Build dedicated MCP server implementation +- **Week 3**: Deploy and configure server infrastructure +- **Week 4**: Set up monitoring, logging, and scaling +- **Week 5**: Handle production issues and debugging +- **Ongoing**: Server maintenance, updates, and operations -## Feature Comparison +### Risk Assessment -| Feature | MCP | UTCP | -|---------|-----|------| -| **Tool Discovery** | Via MCP Server | Via manual discovery endpoint | -| **Protocol Support** | HTTP Streaming | HTTP, WebSockets, gRPC, CLI, etc. | -| **Authentication** | Handled by MCP Server | Uses tool's native authentication | -| **Streaming** | Native support | Supported via appropriate communication protocol (SSE, WebSockets) | -| **Implementation Complexity** | High (requires wrapper servers) | Low (simple JSON definitions) | -| **Performance** | Additional overhead due to proxy | Direct, native performance | -| **Evolution Speed** | Slow (all participants must update) | Fast (individual communication protocols can evolve independently) | +| Risk Factor | UTCP | MCP | +|-------------|------|-----| +| **Single Point of Failure** | None (direct calls) | MCP servers | +| **Vendor Lock-in** | Low (standard protocols) | Medium (MCP-specific) | +| **Maintenance Burden** | Low | High | +| **Security Surface** | Minimal | Expanded | +| **Performance Risk** | Low | Medium | -## When to Choose Each Protocol - -### Choose MCP When: - -- You need strict standardization across all tools -- You're building a closed ecosystem where you control all components -- You're willing to invest in building and maintaining wrapper servers +## Decision Framework ### Choose UTCP When: -- You want to leverage existing APIs without building wrappers -- You need to support diverse communication protocols -- You value direct, efficient communication -- You prioritize low implementation overhead -- You want to minimize infrastructure costs +✅ **You have existing APIs** that work well +✅ **You want minimal infrastructure** overhead +✅ **You need multiple protocols** (HTTP, WebSocket, CLI, etc.) +✅ **You prioritize performance** and direct communication +✅ **You want to leverage existing** auth, monitoring, scaling +✅ **You have limited resources** for server maintenance +✅ **You need rapid deployment** of AI tool access + +### Choose MCP When: -## Real-World Example +✅ **You need strict protocol standardization** across all tools +✅ **You're building a closed ecosystem** with full control +✅ **You have resources** for building and maintaining servers +✅ **You need MCP-specific features** like resources and prompts +✅ **You're already invested** in MCP infrastructure +✅ **You prefer centralized control** over tool access + +### Hybrid Approach When: + +✅ **You're migrating from MCP** to UTCP gradually +✅ **You have mixed requirements** (some tools need MCP features) +✅ **You want to evaluate both** approaches in production +✅ **You have legacy MCP investments** to preserve + +## Real-World Examples + +### E-commerce API Integration + +#### UTCP Approach + +**E-commerce API with UTCP:** +- Keep existing product API endpoints unchanged (GET /products/\{product_id\}) +- Add single UTCP discovery endpoint (GET /utcp) +- Return UTCP manual describing available tools and how to call them +- Tools directly reference existing API endpoints with proper parameters +- Total additional code: ~10 lines +- Additional infrastructure: 0 servers + +#### MCP Approach + +**E-commerce API with MCP:** +- Requires building dedicated MCP server wrapper +- Implement tool listing and calling handlers in MCP server +- MCP server calls existing API endpoints on behalf of clients +- Additional server deployment, monitoring, and scaling required +- Total additional code: ~50+ lines +- Additional infrastructure: 1+ servers + +### Database Query Tool + +#### UTCP Approach +```json +{ + "name": "query_database", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.company.com/query", + "http_method": "POST", + "body": {"sql": "${query}"}, + "auth": { + "auth_type": "api_key", + "api_key": "${DB_API_KEY}", + "var_name": "Authorization", + "location": "header" + } + } +} +``` + +#### MCP Approach +**MCP database approach:** +- Requires MCP server with database connection management +- Connection pooling, query validation, and security in MCP server +- Much more complex implementation than direct database access +- Additional abstraction layer between client and database + +## Performance Benchmarks + +### Latency Comparison + +| Scenario | UTCP | MCP | Difference | +|----------|------|-----|------------| +| Simple API call | 50ms | 75ms | +50% overhead | +| Complex query | 200ms | 250ms | +25% overhead | +| File operation | 10ms | 20ms | +100% overhead | +| Streaming data | Real-time | Buffered | Significant delay | + +### Resource Usage + +| Resource | UTCP | MCP | +|----------|------|-----| +| Memory | 0MB (no servers) | 50-200MB per server | +| CPU | 0% (no processing) | 5-15% per server | +| Network | Direct | Double hops | +| Storage | 0GB | Logs, state, config | + +## Migration Timeline + +### From MCP to UTCP + +**Phase 1 (Week 1): Assessment** +- Inventory existing MCP servers +- Identify high-value tools for migration +- Plan migration strategy + +**Phase 2 (Week 2-3): Hybrid Setup** +- Install UTCP with MCP plugin +- Test existing MCP tools through UTCP +- Validate functionality + +**Phase 3 (Week 4-8): Gradual Migration** +- Migrate tools one by one to native UTCP +- Add `/utcp` endpoints to existing APIs +- Update client configurations + +**Phase 4 (Week 9+): Cleanup** +- Deprecate MCP servers +- Remove MCP infrastructure +- Monitor and optimize + +[**Detailed migration guide →**](./migration-v0.1-to-v1.0.md) + +## Community & Ecosystem + +### UTCP Ecosystem +- **Multiple language implementations**: Python, TypeScript, Go, Rust +- **Growing protocol support**: HTTP, WebSocket, CLI, SSE, Text, MCP +- **Active development**: Regular releases and improvements +- **Open governance**: RFC process for changes + +### MCP Ecosystem +- **Anthropic-led development**: Centralized development +- **Growing tool library**: Community-contributed servers +- **IDE integrations**: Claude Desktop, Cline, etc. +- **Established patterns**: Well-documented server patterns -Consider an organization with an existing REST API that they want to expose to AI agents: +## Conclusion -**With MCP:** -1. Build an MCP server that wraps the REST API -2. Translate all calls between MCP format and REST format -3. Maintain and scale this additional server infrastructure -4. Handle authentication translation between MCP and the API +Both UTCP and MCP solve the tool integration problem, but with fundamentally different approaches: -**With UTCP:** -1. Create a simple JSON definition describing the REST API -2. Expose this definition via a discovery endpoint (typically `/utcp`) -3. The AI agent can now call the REST API directly +**UTCP excels when you:** +- Want to leverage existing APIs without additional infrastructure +- Need support for multiple communication protocols +- Prioritize performance and direct communication +- Have limited resources for server maintenance +- Want rapid deployment and minimal complexity -## Code comparison +**MCP excels when you:** +- Need strict protocol standardization +- Are building a controlled ecosystem +- Have resources for server infrastructure +- Need MCP-specific features beyond tool calling +- Prefer centralized tool management -You can find a full typescript example detailing the MCP and UTCP approach [here](https://github.com/universal-tool-calling-protocol/typescript-utcp/tree/main/examples/src/concrete_example). +**For most organizations**, UTCP's "manual" approach offers significant advantages in terms of simplicity, performance, and cost-effectiveness. The ability to expose existing APIs to AI agents with minimal changes and no additional infrastructure makes it an attractive choice for rapid AI tool integration. -## Conclusion +**For gradual adoption**, consider starting with UTCP's MCP plugin to use existing MCP servers while migrating high-value tools to native UTCP protocols over time. -Both MCP and UTCP aim to solve the problem of standardizing tool calling for AI agents, but they take fundamentally different approaches. +## Next Steps -MCP acts as a middleman, requiring all tools to be wrapped in its protocol. This provides standardization but at the cost of additional infrastructure and development overhead. +### To Get Started with UTCP: +1. **[Read the implementation guide](./implementation.md)** - Learn how to implement UTCP +2. **[Choose your protocols](./protocols/index.md)** - Select communication methods +3. **[Check examples](https://github.com/universal-tool-calling-protocol)** - See real implementations across multiple languages -UTCP acts as a manual, describing how to call tools directly using their native interfaces. This eliminates the wrapper tax and leverages existing infrastructure, at the cost of requiring clients to handle different communication protocols. +### To Migrate from MCP: +1. **[Read the MCP integration guide](./protocols/mcp.md)** - Use MCP tools via UTCP +2. **[Plan your migration](./migration-v0.1-to-v1.0.md)** - Step-by-step migration process +3. **[Join the community](https://discord.gg/ZpMbQ8jRbD)** - Get migration support -The choice between them depends on your specific requirements, existing infrastructure, and development resources. However, UTCP's "manual" approach offers significant advantages in terms of simplicity, efficiency, and leveraging existing investments in API infrastructure. +### To Learn More: +- **[UTCP Architecture](./api/index.md)** - Technical deep dive +- **[Security Considerations](./security.md)** - Security best practices +- **[Tool Provider Guide](./for-tool-providers.md)** - Expose your tools diff --git a/sidebars.ts b/sidebars.ts index 2897139..ea4924a 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -13,21 +13,8 @@ import type {SidebarsConfig} from '@docusaurus/plugin-content-docs'; Create as many sidebars as you want. */ const sidebars: SidebarsConfig = { - // By default, Docusaurus generates a sidebar from the docs folder structure + // Main documentation sidebar tutorialSidebar: [{type: 'autogenerated', dirName: '.'}], - - // But you can create a sidebar manually - /* - tutorialSidebar: [ - 'intro', - 'hello', - { - type: 'category', - label: 'Tutorial', - items: ['tutorial-basics/create-a-document'], - }, - ], - */ }; export default sidebars; diff --git a/src/data/contributors.json b/src/data/contributors.json index c379760..08a952f 100644 --- a/src/data/contributors.json +++ b/src/data/contributors.json @@ -1,14 +1,14 @@ { - "generated_at": "2025-09-07T06:35:04.854Z", + "generated_at": "2025-09-05T06:38:23.414Z", "total_contributors": 11, - "total_contributions": 543, - "total_impact_score": 397, - "total_recent_activity": 399, + "total_contributions": 521, + "total_impact_score": 375, + "total_recent_activity": 377, "scoring_method": "simplified_recent_activity", - "total_additions": 326249, - "total_deletions": 78919, - "total_changes": 405168, - "total_commits_analyzed": 543, + "total_additions": 321522, + "total_deletions": 78207, + "total_changes": 399729, + "total_commits_analyzed": 521, "contributors": [ { "id": 43811028, @@ -25,7 +25,7 @@ "followers": 22, "following": 2, "created_at": "2018-10-03T10:10:15Z", - "contributions": 188, + "contributions": 179, "repositories": [ "python-utcp", "utcp-specification", @@ -38,18 +38,18 @@ "utcp-agent" ], "repo_count": 9, - "impact_score": 188, - "total_prs": 176, - "total_merged_prs": 141, - "total_recent_commits": 188, - "total_commits": 188, + "impact_score": 179, + "total_prs": 169, + "total_merged_prs": 136, + "total_recent_commits": 179, + "total_commits": 179, "total_reviews": 9, - "last_activity": "2025-09-05T14:26:56Z", + "last_activity": "2025-09-04T12:36:01Z", "pr_success_rate": 80, - "total_additions": 214048, - "total_deletions": 54540, - "total_changes": 268588, - "commits_analyzed": 188 + "total_additions": 213252, + "total_deletions": 54522, + "total_changes": 267774, + "commits_analyzed": 179 }, { "id": 170965471, @@ -63,8 +63,8 @@ "blog": null, "hireable": true, "public_repos": 28, - "followers": 30, - "following": 64, + "followers": 31, + "following": 59, "created_at": "2024-05-27T16:22:55Z", "contributions": 244, "repositories": [ @@ -96,30 +96,29 @@ "blog": null, "hireable": false, "public_repos": 17, - "followers": 4, - "following": 4, + "followers": 3, + "following": 3, "created_at": "2020-08-11T00:28:15Z", - "contributions": 48, + "contributions": 42, "repositories": [ "python-utcp", "utcp-specification", ".github", - "utcp-mcp", - "utcp-agent" + "utcp-mcp" ], - "repo_count": 5, - "impact_score": 48, - "total_prs": 81, - "total_merged_prs": 67, - "total_recent_commits": 48, - "total_commits": 48, + "repo_count": 4, + "impact_score": 42, + "total_prs": 70, + "total_merged_prs": 60, + "total_recent_commits": 42, + "total_commits": 42, "total_reviews": 0, - "last_activity": "2025-09-05T09:36:48Z", - "pr_success_rate": 83, - "total_additions": 18612, - "total_deletions": 5610, - "total_changes": 24222, - "commits_analyzed": 48 + "last_activity": "2025-09-04T11:01:20Z", + "pr_success_rate": 86, + "total_additions": 17408, + "total_deletions": 5368, + "total_changes": 22776, + "commits_analyzed": 42 }, { "id": 5594869, @@ -132,11 +131,11 @@ "location": null, "blog": null, "hireable": false, - "public_repos": 66, - "followers": 7, + "public_repos": 65, + "followers": 5, "following": 1, "created_at": "2013-10-02T14:02:46Z", - "contributions": 25, + "contributions": 24, "repositories": [ "python-utcp", "utcp-mcp", @@ -144,18 +143,18 @@ "langchain-utcp-adapters" ], "repo_count": 4, - "impact_score": 25, - "total_prs": 58, - "total_merged_prs": 46, - "total_recent_commits": 25, - "total_commits": 25, + "impact_score": 24, + "total_prs": 55, + "total_merged_prs": 44, + "total_recent_commits": 24, + "total_commits": 24, "total_reviews": 0, - "last_activity": "2025-09-05T14:45:00Z", - "pr_success_rate": 79, - "total_additions": 12125, - "total_deletions": 3953, - "total_changes": 16078, - "commits_analyzed": 25 + "last_activity": "2025-09-02T14:17:44Z", + "pr_success_rate": 80, + "total_additions": 12023, + "total_deletions": 3946, + "total_changes": 15969, + "commits_analyzed": 24 }, { "id": 7006124, @@ -169,26 +168,26 @@ "blog": null, "hireable": false, "public_repos": 26, - "followers": 7, + "followers": 6, "following": 0, "created_at": "2014-03-20T02:28:28Z", - "contributions": 15, + "contributions": 11, "repositories": [ "typescript-utcp-sdk" ], "repo_count": 1, - "impact_score": 15, + "impact_score": 11, "total_prs": 0, "total_merged_prs": 0, - "total_recent_commits": 15, - "total_commits": 15, + "total_recent_commits": 11, + "total_commits": 11, "total_reviews": 0, - "last_activity": "2025-09-06T12:42:01Z", + "last_activity": "2025-08-31T02:56:43Z", "pr_success_rate": 0, - "total_additions": 10926, - "total_deletions": 2054, - "total_changes": 12980, - "commits_analyzed": 15 + "total_additions": 8399, + "total_deletions": 1706, + "total_changes": 10105, + "commits_analyzed": 11 }, { "id": 21357398, @@ -202,7 +201,7 @@ "blog": null, "hireable": false, "public_repos": 18, - "followers": 3, + "followers": 2, "following": 1, "created_at": "2016-08-31T08:24:11Z", "contributions": 7, @@ -213,13 +212,13 @@ ], "repo_count": 3, "impact_score": 7, - "total_prs": 55, - "total_merged_prs": 45, + "total_prs": 52, + "total_merged_prs": 43, "total_recent_commits": 7, "total_commits": 7, "total_reviews": 5, "last_activity": "2025-09-02T14:17:22Z", - "pr_success_rate": 82, + "pr_success_rate": 83, "total_additions": 28358, "total_deletions": 916, "total_changes": 29274, @@ -237,7 +236,7 @@ "blog": null, "hireable": false, "public_repos": 13, - "followers": 17, + "followers": 15, "following": 13, "created_at": "2019-01-07T15:20:48Z", "contributions": 6, @@ -247,51 +246,18 @@ ], "repo_count": 2, "impact_score": 5, - "total_prs": 51, - "total_merged_prs": 39, + "total_prs": 48, + "total_merged_prs": 38, "total_recent_commits": 6, "total_commits": 6, "total_reviews": 0, "last_activity": "2025-07-31T09:55:48Z", - "pr_success_rate": 76, + "pr_success_rate": 79, "total_additions": 2282, "total_deletions": 309, "total_changes": 2591, "commits_analyzed": 6 }, - { - "id": 65916846, - "login": "actions-user", - "name": "actions-user", - "avatar_url": "https://avatars.githubusercontent.com/u/65916846?v=4", - "html_url": "https://github.com/actions-user", - "bio": null, - "company": "@actions", - "location": null, - "blog": "https://github.com/actions", - "hireable": false, - "public_repos": 0, - "followers": 3246, - "following": 0, - "created_at": "2020-05-25T17:35:50Z", - "contributions": 4, - "repositories": [ - "utcp-specification" - ], - "repo_count": 1, - "impact_score": 4, - "total_prs": 26, - "total_merged_prs": 25, - "total_recent_commits": 4, - "total_commits": 4, - "total_reviews": 0, - "last_activity": "2025-09-06T06:34:17Z", - "pr_success_rate": 96, - "total_additions": 184, - "total_deletions": 150, - "total_changes": 334, - "commits_analyzed": 4 - }, { "id": 150209392, "login": "bruce-gene", @@ -313,18 +279,51 @@ ], "repo_count": 1, "impact_score": 3, - "total_prs": 26, - "total_merged_prs": 25, + "total_prs": 22, + "total_merged_prs": 22, "total_recent_commits": 4, "total_commits": 4, "total_reviews": 0, "last_activity": "2025-08-01T15:02:36Z", - "pr_success_rate": 96, + "pr_success_rate": 100, "total_additions": 7, "total_deletions": 7, "total_changes": 14, "commits_analyzed": 4 }, + { + "id": 65916846, + "login": "actions-user", + "name": "actions-user", + "avatar_url": "https://avatars.githubusercontent.com/u/65916846?v=4", + "html_url": "https://github.com/actions-user", + "bio": null, + "company": "@actions", + "location": null, + "blog": "https://github.com/actions", + "hireable": false, + "public_repos": 0, + "followers": 3235, + "following": 0, + "created_at": "2020-05-25T17:35:50Z", + "contributions": 2, + "repositories": [ + "utcp-specification" + ], + "repo_count": 1, + "impact_score": 2, + "total_prs": 22, + "total_merged_prs": 22, + "total_recent_commits": 2, + "total_commits": 2, + "total_reviews": 0, + "last_activity": "2025-09-04T13:03:07Z", + "pr_success_rate": 100, + "total_additions": 86, + "total_deletions": 53, + "total_changes": 139, + "commits_analyzed": 2 + }, { "id": 210855846, "login": "aliraza1006", @@ -346,7 +345,7 @@ ], "repo_count": 1, "impact_score": 1, - "total_prs": 1, + "total_prs": 0, "total_merged_prs": 0, "total_recent_commits": 1, "total_commits": 1, @@ -380,12 +379,12 @@ "repo_count": 1, "impact_score": 1, "total_prs": 7, - "total_merged_prs": 7, + "total_merged_prs": 6, "total_recent_commits": 1, "total_commits": 1, "total_reviews": 0, "last_activity": "2025-07-24T16:29:37Z", - "pr_success_rate": 100, + "pr_success_rate": 86, "total_additions": 318, "total_deletions": 0, "total_changes": 318, diff --git a/versioned_docs/version-1.0/api/core/utcp/exceptions/utcp_serializer_validation_error.md b/versioned_docs/version-1.0/api/core/utcp/exceptions/utcp_serializer_validation_error.md index 4dcaaec..378f221 100644 --- a/versioned_docs/version-1.0/api/core/utcp/exceptions/utcp_serializer_validation_error.md +++ b/versioned_docs/version-1.0/api/core/utcp/exceptions/utcp_serializer_validation_error.md @@ -9,6 +9,20 @@ sidebar_label: utcp_serializer_validation_error ### class UtcpSerializerValidationError {#utcpserializervalidationerror} -*No class documentation available* +
+Documentation + +Exception raised when a serializer validation fails. + +Thrown by serializers when they cannot validate or convert data structures +due to invalid format, missing required fields, or type mismatches. +Contains the original validation error details for debugging. + + +**Usage** + +Typically caught when loading configuration files or processing +external data that doesn't conform to UTCP specifications. +
--- diff --git a/versioned_docs/version-1.0/api/core/utcp/implementations/post_processors/filter_dict_post_processor.md b/versioned_docs/version-1.0/api/core/utcp/implementations/post_processors/filter_dict_post_processor.md new file mode 100644 index 0000000..6193415 --- /dev/null +++ b/versioned_docs/version-1.0/api/core/utcp/implementations/post_processors/filter_dict_post_processor.md @@ -0,0 +1,49 @@ +--- +title: filter_dict_post_processor +sidebar_label: filter_dict_post_processor +--- + +# filter_dict_post_processor + +**File:** `core/src/utcp/implementations/post_processors/filter_dict_post_processor.py` + +### class FilterDictPostProcessor ([ToolPostProcessor](./../../interfaces/tool_post_processor.md#toolpostprocessor)) {#filterdictpostprocessor} + +
+Documentation + +Post-processor that filters dictionary keys from tool results. + +Provides flexible filtering capabilities to include or exclude specific keys +from dictionary results, with support for nested dictionaries and lists. +Can be configured to apply filtering only to specific tools or manuals. + + +**Attributes** + +- **`tool_post_processor_type`**: Always "filter_dict" for this processor. +- **`exclude_keys`**: List of keys to remove from dictionary results. +- **`only_include_keys`**: List of keys to keep in dictionary results (all others removed). +- **`exclude_tools`**: List of tool names to skip processing for. +- **`only_include_tools`**: List of tool names to process (all others skipped). +- **`exclude_manuals`**: List of manual names to skip processing for. +- **`only_include_manuals`**: List of manual names to process (all others skipped). +
+ +#### Fields: + +- tool_post_processor_type: Literal['filter_dict'] +- exclude_keys: Optional[List[str]] +- only_include_keys: Optional[List[str]] +- exclude_tools: Optional[List[str]] +- only_include_tools: Optional[List[str]] +- exclude_manuals: Optional[List[str]] +- only_include_manuals: Optional[List[str]] + +--- + +### class FilterDictPostProcessorConfigSerializer ([Serializer](./../../interfaces/serializer.md#serializer)[FilterDictPostProcessor]) {#filterdictpostprocessorconfigserializer} + +*No class documentation available* + +--- diff --git a/versioned_docs/version-1.0/api/core/utcp/implementations/post_processors/limit_strings_post_processor.md b/versioned_docs/version-1.0/api/core/utcp/implementations/post_processors/limit_strings_post_processor.md new file mode 100644 index 0000000..f524d48 --- /dev/null +++ b/versioned_docs/version-1.0/api/core/utcp/implementations/post_processors/limit_strings_post_processor.md @@ -0,0 +1,48 @@ +--- +title: limit_strings_post_processor +sidebar_label: limit_strings_post_processor +--- + +# limit_strings_post_processor + +**File:** `core/src/utcp/implementations/post_processors/limit_strings_post_processor.py` + +### class LimitStringsPostProcessor ([ToolPostProcessor](./../../interfaces/tool_post_processor.md#toolpostprocessor)) {#limitstringspostprocessor} + +
+Documentation + +Post-processor that limits the length of string values in tool results. + +Truncates string values to a specified maximum length to prevent +excessively large responses. Processes nested dictionaries and lists +recursively. Can be configured to apply limiting only to specific +tools or manuals. + + +**Attributes** + +- **`tool_post_processor_type`**: Always "limit_strings" for this processor. +- **`limit`**: Maximum length for string values (default: 10000 characters). +- **`exclude_tools`**: List of tool names to skip processing for. +- **`only_include_tools`**: List of tool names to process (all others skipped). +- **`exclude_manuals`**: List of manual names to skip processing for. +- **`only_include_manuals`**: List of manual names to process (all others skipped). +
+ +#### Fields: + +- tool_post_processor_type: Literal['limit_strings'] +- limit: int +- exclude_tools: Optional[List[str]] +- only_include_tools: Optional[List[str]] +- exclude_manuals: Optional[List[str]] +- only_include_manuals: Optional[List[str]] + +--- + +### class LimitStringsPostProcessorConfigSerializer ([Serializer](./../../interfaces/serializer.md#serializer)[LimitStringsPostProcessor]) {#limitstringspostprocessorconfigserializer} + +*No class documentation available* + +--- diff --git a/versioned_docs/version-1.0/api/core/utcp/implementations/tag_search.md b/versioned_docs/version-1.0/api/core/utcp/implementations/tag_search.md index af032ab..558045e 100644 --- a/versioned_docs/version-1.0/api/core/utcp/implementations/tag_search.md +++ b/versioned_docs/version-1.0/api/core/utcp/implementations/tag_search.md @@ -9,7 +9,38 @@ sidebar_label: tag_search ### class TagAndDescriptionWordMatchStrategy ([ToolSearchStrategy](./../interfaces/tool_search_strategy.md#toolsearchstrategy)) {#taganddescriptionwordmatchstrategy} -*No class documentation available* +
+Documentation + +Tag and description word match strategy. + + +**Implements A Weighted Scoring System That Matches Tools Based On** + +1. Tag matches (higher weight) +2. Description word matches (lower weight) + +The strategy normalizes queries to lowercase, extracts words using regex, +and calculates relevance scores for each tool. Results are sorted by +score in descending order. + + + +**Attributes** + +- **`tool_search_strategy_type`**: Always "tag_and_description_word_match". +- **`description_weight`**: Weight multiplier for description word matches (default: 1.0). +- **`tag_weight`**: Weight multiplier for tag matches (default: 3.0). + + + +**Scoring Algorithm** + +- Each matching tag contributes tag_weight points +- Each matching description word contributes description_weight points +- Tools with higher scores are ranked first +- Tools with zero score are included in results (ranked last) +
#### Fields: diff --git a/versioned_docs/version-1.0/api/core/utcp/plugins/plugin_loader.md b/versioned_docs/version-1.0/api/core/utcp/plugins/plugin_loader.md index 8e8e9c1..3ac6524 100644 --- a/versioned_docs/version-1.0/api/core/utcp/plugins/plugin_loader.md +++ b/versioned_docs/version-1.0/api/core/utcp/plugins/plugin_loader.md @@ -7,6 +7,12 @@ sidebar_label: plugin_loader **File:** `core/src/utcp/plugins/plugin_loader.py` +### Function _load_plugins() {#_load_plugins} + +*No function documentation available* + +--- + ### Function ensure_plugins_initialized() {#ensure_plugins_initialized} *No function documentation available* diff --git a/versioned_docs/version-1.0/api/index.md b/versioned_docs/version-1.0/api/index.md index 51585f8..50ef8a5 100644 --- a/versioned_docs/version-1.0/api/index.md +++ b/versioned_docs/version-1.0/api/index.md @@ -11,8 +11,8 @@ This specification is organized by module of the reference python implementation **Note:** The modules don't have to be implemented in the same way as in the reference implementation, but all of the functionality here needs to be provided. -**Total documented items:** 189 -**Modules documented:** 39 +**Total documented items:** 195 +**Modules documented:** 41 ## Core Modules @@ -93,6 +93,16 @@ Core UTCP framework components that define the fundamental interfaces and implem - **Contains:** 2 classes, 12 methods +### [utcp.implementations.post_processors.filter_dict_post_processor](./core\utcp\implementations\post_processors\filter_dict_post_processor.md) + +- **Contains:** 2 classes + + +### [utcp.implementations.post_processors.limit_strings_post_processor](./core\utcp\implementations\post_processors\limit_strings_post_processor.md) + +- **Contains:** 2 classes + + ### [utcp.implementations.tag_search](./core\utcp\implementations\tag_search.md) - **Contains:** 2 classes, 3 methods @@ -140,7 +150,7 @@ Core UTCP framework components that define the fundamental interfaces and implem ### [utcp.plugins.plugin_loader](./core\utcp\plugins\plugin_loader.md) -- **Contains:** 1 functions +- **Contains:** 2 functions ### [utcp.utcp_client](./core\utcp\utcp_client.md) @@ -154,7 +164,7 @@ Plugin implementations that extend UTCP with specific transport protocols and ca ### [communication_protocols.cli.src.utcp_cli.cli_call_template](./plugins\communication_protocols\cli\src\utcp_cli\cli_call_template.md) -- **Contains:** 2 classes, 2 methods +- **Contains:** 3 classes, 2 methods ### [communication_protocols.cli.src.utcp_cli.cli_communication_protocol](./plugins\communication_protocols\cli\src\utcp_cli\cli_communication_protocol.md) diff --git a/versioned_docs/version-1.0/api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md b/versioned_docs/version-1.0/api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md index abfe4aa..54c7dca 100644 --- a/versioned_docs/version-1.0/api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md +++ b/versioned_docs/version-1.0/api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md @@ -7,30 +7,173 @@ sidebar_label: cli_call_template **File:** `plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py` +### class CommandStep {#commandstep} + +
+Documentation + +Configuration for a single command step in a CLI execution flow. + + +**Attributes** + +- **`command`**: The command string to execute. Can contain UTCP_ARG_argname_UTCP_END + placeholders that will be replaced with values from tool_args. Can also + reference previous command outputs using $CMD_0_OUTPUT, $CMD_1_OUTPUT, etc. +- **`append_to_final_output`**: Whether this command's output should be included + in the final result. If not specified, defaults to False for all + commands except the last one. + + + +**Basic Command Step** + +```json + { + "command": "git status", + "append_to_final_output": true + } +``` + + + +**Command With Argument Placeholders And Output Reference** + +```json + { + "command": "echo "Cloning to: UTCP_ARG_target_dir_UTCP_END, previous status: $CMD_0_OUTPUT"", + "append_to_final_output": true + } +``` +
+ +#### Fields: + +- command: str +- append_to_final_output: Optional[bool] + +--- + ### class CliCallTemplate ([CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) {#clicalltemplate}
Documentation -Call template configuration for Command Line Interface tools. +Call template configuration for Command Line Interface (CLI) tools. + +This class defines the configuration for executing command-line tools and +programs as UTCP tool providers. Commands are executed in a single subprocess +to maintain state (like directory changes) between commands. + + + + +**You Can Reference The Output Of Previous Commands Using Variables** + + +- **`Example`**: `echo "Previous result: $CMD_0_OUTPUT"` -Enables execution of command-line tools and programs as UTCP providers. -Supports environment variable injection and custom working directories. **Attributes** -- **`call_template_type`**: Always "cli" for CLI providers. -- **`command_name`**: The name or path of the command to execute. -- **`env_vars`**: Optional environment variables to set during command execution. -- **`working_dir`**: Optional custom working directory for command execution. -- **`auth`**: Always None - CLI providers don't support authentication. +- **`call_template_type`**: The type of the call template. Must be "cli". +- **`commands`**: A list of CommandStep objects defining the commands to execute + in order. Each command can contain UTCP_ARG_argname_UTCP_END placeholders + that will be replaced with values from tool_args during execution. +- **`env_vars`**: A dictionary of environment variables to set for the command's + execution context. Values can be static strings or placeholders for + variables from the UTCP client's variable substitutor. +- **`working_dir`**: The working directory from which to run the commands. If not + provided, it defaults to the current process's working directory. +- **`auth`**: Authentication details. Not applicable to the CLI protocol, so it + is always None. + + + +**Cross-Platform Directory Operations** + +```json + { + "name": "cross_platform_dir_tool", + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_target_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "ls -la", + "append_to_final_output": true + } + ] + } +``` + + + +**Referencing Previous Command Output** + +```json + { + "name": "reference_previous_output_tool", + "call_template_type": "cli", + "commands": [ + { + "command": "git status --porcelain", + "append_to_final_output": false + }, + { + "command": "echo "Found changes: $CMD_0_OUTPUT"", + "append_to_final_output": true + } + ] + } +``` + + + +**Command With Environment Variables And Placeholders** + +```json + { + "name": "python_multi_step_tool", + "call_template_type": "cli", + "commands": [ + { + "command": "python setup.py install", + "append_to_final_output": false + }, + { + "command": "python script.py --input UTCP_ARG_input_file_UTCP_END --result "$CMD_0_OUTPUT"" + } + ], + "env_vars": { + "PYTHONPATH": "/custom/path", + "API_KEY": "${API_KEY_VAR}" + } + } +``` + + + +**Security Considerations** + +- Commands are executed in a subprocess. Ensure that the commands +specified are from a trusted source. +- Avoid passing unsanitized user input directly into the command string. +Use tool argument validation where possible. +- All placeholders are replaced with string values from tool_args. +- Commands should use the appropriate syntax for the target platform +(PowerShell on Windows, Bash on Unix). +- Previous command outputs are available as variables but should be +used carefully to avoid command injection.
#### Fields: - call_template_type: Literal['cli'] -- command_name: str +- commands: List[CommandStep] - env_vars: Optional[Dict[str, str]] - working_dir: Optional[str] - auth: None @@ -46,13 +189,42 @@ Supports environment variable injection and custom working directories.
to_dict(self, obj: CliCallTemplate) -> dict -*No method documentation available* +Converts a `CliCallTemplate` instance to its dictionary representation. + + +**Args** + +- **`obj`**: The `CliCallTemplate` instance to serialize. + + + +**Returns** + +A dictionary representing the `CliCallTemplate`.
validate_dict(self, obj: dict) -> CliCallTemplate -*No method documentation available* +Validates a dictionary and constructs a `CliCallTemplate` instance. + + +**Args** + +- **`obj`**: The dictionary to validate and deserialize. + + + +**Returns** + +A `CliCallTemplate` instance. + + + +**Raises** + +- **`[UtcpSerializerValidationError](./../../../../../core/utcp/exceptions/utcp_serializer_validation_error.md#utcpserializervalidationerror)`**: If the dictionary is not a valid + representation of a `CliCallTemplate`.
--- diff --git a/versioned_docs/version-1.0/api/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.md b/versioned_docs/version-1.0/api/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.md index ee2e071..5ed1941 100644 --- a/versioned_docs/version-1.0/api/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.md +++ b/versioned_docs/version-1.0/api/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.md @@ -9,87 +9,98 @@ sidebar_label: cli_communication_protocol ### class CliCommunicationProtocol ([CommunicationProtocol](./../../../../../core/utcp/interfaces/communication_protocol.md#communicationprotocol)) {#clicommunicationprotocol} -
-Documentation - -Transport implementation for CLI-based tool providers. +*No class documentation available* -Handles communication with command-line tools by executing processes -and managing their input/output. Supports both tool discovery and -execution phases with comprehensive error handling and timeout management. +#### Methods: +
+async register_manual(self, caller, manual_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> [RegisterManualResult](./../../../../../core/utcp/data/register_manual_response.md#registermanualresult) -**Features** +Registers a CLI-based manual and discovers its tools. -- Asynchronous subprocess execution with proper cleanup -- [Tool](./../../../../../core/utcp/data/tool.md#tool) discovery through startup commands returning UTCP manuals -- Flexible argument formatting for various CLI conventions -- Environment variable injection for authentication -- JSON output parsing with graceful fallback to text -- Cross-platform command parsing and execution -- Configurable working directories and timeouts -- Process lifecycle management with proper termination +This method executes the command specified in the `[CliCallTemplate](./cli_call_template.md#clicalltemplate)`'s +`command_name` field. It then attempts to parse the command's output +(stdout) as a UTCP manual in JSON format. +**Args** -**Architecture** +- **`caller`**: The UTCP client instance that is calling this method. +- **`manual_call_template`**: The `[CliCallTemplate](./cli_call_template.md#clicalltemplate)` containing the details for + tool discovery, such as the command to run. -CLI tools are discovered by executing the provider's command_name -and parsing the output for UTCP manual JSON. [Tool](./../../../../../core/utcp/data/tool.md#tool) calls execute -the same command with formatted arguments and return processed output. +**Returns** -**Attributes** +A `[RegisterManualResult](./../../../../../core/utcp/data/register_manual_response.md#registermanualresult)` object indicating whether the registration +was successful and containing the discovered tools. -- **`_log`**: Logger function for debugging and error reporting. -
-#### Methods: -
-async register_manual(self, caller, manual_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> [RegisterManualResult](./../../../../../core/utcp/data/register_manual_response.md#registermanualresult) +**Raises** -*No method documentation available* +- **`ValueError`**: If the `manual_call_template` is not an instance of + `[CliCallTemplate](./cli_call_template.md#clicalltemplate)` or if `command_name` is not set.
async deregister_manual(self, caller, manual_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> None -*No method documentation available* +Deregisters a CLI manual. + +For the CLI protocol, this is a no-op as there are no persistent +connections to terminate. + + +**Args** + +- **`caller`**: The UTCP client instance that is calling this method. +- **`manual_call_template`**: The call template of the manual to deregister.
async call_tool(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> Any -Call a CLI tool. +Calls a CLI tool by executing its command. -Executes the command specified by provider.command_name with the provided arguments. +This method constructs and executes the command specified in the +`[CliCallTemplate](./cli_call_template.md#clicalltemplate)`. It formats the provided `tool_args` as command-line +arguments and runs the command in a subprocess. **Args** -- **`caller`**: The UTCP client that is calling this method. -- **`tool_name`**: Name of the tool to call -- **`tool_args`**: Arguments for the tool call -- **`tool_call_template`**: The [CliCallTemplate](./cli_call_template.md#clicalltemplate) for the tool +- **`caller`**: The UTCP client instance that is calling this method. +- **`tool_name`**: The name of the tool to call. +- **`tool_args`**: A dictionary of arguments for the tool call. +- **`tool_call_template`**: The `[CliCallTemplate](./cli_call_template.md#clicalltemplate)` for the tool. -**The Output From The Command Execution Based On Exit Code** +**Returns** +The result of the command execution. If the command exits with a code +of 0, it returns the content of stdout. If the exit code is non-zero, +it returns the content of stderr. **Raises** -- **`ValueError`**: If provider is not a CliProvider or command_name is not set +- **`ValueError`**: If `tool_call_template` is not an instance of + `[CliCallTemplate](./cli_call_template.md#clicalltemplate)` or if `command_name` is not set.
async call_tool_streaming(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: [CallTemplate](./../../../../../core/utcp/data/call_template.md#calltemplate)) -> AsyncGenerator[Any, None] -*No method documentation available* +Streaming calls are not supported for the CLI protocol. + + +**Raises** + +- **`NotImplementedError`**: Always, as this functionality is not supported.
--- diff --git a/versioned_docs/version-1.0/api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md b/versioned_docs/version-1.0/api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md index 2420d59..98dda8c 100644 --- a/versioned_docs/version-1.0/api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md +++ b/versioned_docs/version-1.0/api/plugins/communication_protocols/http/src/utcp_http/http_call_template.md @@ -20,6 +20,81 @@ parameters using \{parameter_name\} syntax. All tool arguments not mapped to URL body, headers or query pattern parameters are passed as query parameters using '?arg_name=\{arg_value\}'. +**Basic Http Get Request** + +```json + { + "name": "my_rest_api", + "call_template_type": "http", + "url": "https://api.example.com/users/{user_id}", + "http_method": "GET" + } +``` + + + +**Post With Authentication** + +```json + { + "name": "secure_api", + "call_template_type": "http", + "url": "https://api.example.com/users", + "http_method": "POST", + "content_type": "application/json", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_KEY}", + "var_name": "Authorization", + "location": "header" + }, + "headers": { + "X-Custom-Header": "value" + }, + "body_field": "body", + "header_fields": ["user_id"] + } +``` + + + +**Oauth2 Authentication** + +```json + { + "name": "oauth_api", + "call_template_type": "http", + "url": "https://api.example.com/data", + "http_method": "GET", + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token" + } + } +``` + + + +**Basic Authentication** + +```json + { + "name": "basic_auth_api", + "call_template_type": "http", + "url": "https://api.example.com/secure", + "http_method": "GET", + "auth": { + "auth_type": "basic", + "username": "${USERNAME}", + "password": "${PASSWORD}" + } + } +``` + + + **Attributes** - **`call_template_type`**: Always "http" for HTTP providers. diff --git a/versioned_docs/version-1.0/api/plugins/communication_protocols/http/src/utcp_http/openapi_converter.md b/versioned_docs/version-1.0/api/plugins/communication_protocols/http/src/utcp_http/openapi_converter.md index 0a68759..dd41955 100644 --- a/versioned_docs/version-1.0/api/plugins/communication_protocols/http/src/utcp_http/openapi_converter.md +++ b/versioned_docs/version-1.0/api/plugins/communication_protocols/http/src/utcp_http/openapi_converter.md @@ -22,14 +22,50 @@ a UTCP tool with appropriate input/output schemas. **Features** -- Complete OpenAPI specification parsing -- Recursive JSON reference ($ref) resolution -- Authentication scheme conversion (API key, Basic, OAuth2) -- Input parameter and request body handling -- Response schema extraction -- URL template and path parameter support -- Provider name normalization -- Placeholder variable generation for configuration +- Complete OpenAPI specification parsing. +- Recursive JSON reference ($ref) resolution. +- Authentication scheme conversion (API key, Basic, OAuth2). +- Input parameter and request body handling. +- Response schema extraction. +- URL template and path parameter support. +- Call template name normalization. +- Placeholder variable generation for configuration. + + + +**Basic Openapi Conversion** + +```python + from utcp_http.openapi_converter import OpenApiConverter + + # Assuming you have a method to fetch and parse the spec + openapi_spec = fetch_and_parse_spec("https://api.example.com/openapi.json") + + converter = OpenApiConverter(openapi_spec) + manual = converter.convert() + + # Use the generated manual with a UTCP client + # client = await UtcpClient.create() + # await client.register_manual(manual) +``` + + + +**Converting Local Openapi File** + +```python + import yaml + + converter = OpenApiConverter() + + +**With Open("Api_Spec.Yaml", "R") As F** + +spec_content = yaml.safe_load(f) + +converter = OpenApiConverter(spec_content) +manual = converter.convert() +``` @@ -54,7 +90,16 @@ operation, and creating corresponding UTCP tools with HTTP call_templates.
convert(self) -> [UtcpManual](./../../../../../core/utcp/data/utcp_manual.md#utcpmanual) -*No method documentation available* +Converts the loaded OpenAPI specification into a [UtcpManual](./../../../../../core/utcp/data/utcp_manual.md#utcpmanual). + +This is the main entry point for the conversion process. It iterates through +the paths and operations in the specification, creating a UTCP tool for each +one. + + +**Returns** + +A [UtcpManual](./../../../../../core/utcp/data/utcp_manual.md#utcpmanual) object containing all the tools generated from the spec.
--- diff --git a/versioned_docs/version-1.0/api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md b/versioned_docs/version-1.0/api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md index 2b36fbb..8f5cc77 100644 --- a/versioned_docs/version-1.0/api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md +++ b/versioned_docs/version-1.0/api/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.md @@ -44,12 +44,109 @@ interfaces. Supports both stdio (local process) and HTTP (remote) transport methods. +**Basic Mcp Server With Stdio Transport** + +```json + { + "name": "mcp_server", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "filesystem": { + "command": "node", + "args": ["mcp-server.js"], + "env": {"NODE_ENV": "production"} + } + } + } + } +``` + + + +**Mcp Server With Working Directory** + +```json + { + "name": "mcp_tools", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "tools": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/app/mcp", + "env": { + "PYTHONPATH": "/app", + "LOG_LEVEL": "INFO" + } + } + } + } + } +``` + + + +**Mcp Server With Oauth2 Authentication** + +```json + { + "name": "secure_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "secure_server": { + "transport": "http", + "url": "https://mcp.example.com" + } + } + }, + "auth": { + "auth_type": "oauth2", + "token_url": "https://auth.example.com/token", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "scope": "read:tools" + } + } +``` + + + +**During Migration (Utcp With Mcp)** + +```python + # UTCP Client with MCP plugin + client = await UtcpClient.create() + result = await client.call_tool("filesystem.read_file", { + "path": "/data/file.txt" + }) +``` + + + +**After Migration (Pure Utcp)** + +```python + # UTCP Client with native protocol + client = await UtcpClient.create() + result = await client.call_tool("filesystem.read_file", { + "path": "/data/file.txt" + }) +``` + + + **Attributes** - **`call_template_type`**: Always "mcp" for MCP providers. - **`config`**: Configuration object containing MCP server definitions. This follows the same format as the official MCP server configuration. - **`auth`**: Optional OAuth2 authentication for HTTP-based MCP servers. +- **`register_resources_as_tools`**: Whether to register MCP resources as callable tools. + When True, server resources are exposed as tools that can be called. + Default is False.
#### Fields: @@ -57,6 +154,7 @@ transport methods. - call_template_type: Literal['mcp'] - config: McpConfig - auth: Optional[[OAuth2Auth](./../../../../../core/utcp/data/auth_implementations/oauth2_auth.md#oauth2auth)] +- register_resources_as_tools: bool --- diff --git a/versioned_docs/version-1.0/for-tool-providers.md b/versioned_docs/version-1.0/for-tool-providers.md new file mode 100644 index 0000000..87ff739 --- /dev/null +++ b/versioned_docs/version-1.0/for-tool-providers.md @@ -0,0 +1,580 @@ +--- +id: for-tool-providers +title: For Tool Providers +sidebar_position: 2 +--- + +# For Tool Providers + +:::info Language Note +This guide covers how to expose your existing APIs and services as UTCP tools. UTCP implementations are available in multiple languages - check the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) for Python, TypeScript, Go, and other language implementations. +::: + +This guide helps you expose your tools through UTCP so they can be discovered and used by AI agents and other applications. + +## Overview + +As a tool provider, you'll create a **UTCP Manual** - a standardized description of your tools that tells clients how to call them directly using their native protocols. This eliminates the need for wrapper servers and allows direct communication. + +## Quick Start + +### 1. Create a Simple Manual + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [ + { + "name": "get_user", + "description": "Retrieve user information by ID", + "inputs": { + "type": "object", + "properties": { + "user_id": {"type": "string", "description": "User identifier"} + }, + "required": ["user_id"] + }, + "outputs": { + "type": "object", + "properties": { + "id": {"type": "string"}, + "name": {"type": "string"}, + "email": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users/${user_id}", + "http_method": "GET" + } + } + ] +} +``` + +### 2. Expose via Discovery Endpoint + +Create an HTTP endpoint that returns your UTCP manual: + +**Endpoint**: `GET /utcp` +**Response**: +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [ + // ... your tools here + ] +} +``` + +Your existing API endpoints remain unchanged. For example: +- `GET /users/{user_id}` - Returns user data +- `POST /orders` - Creates new orders +- etc. + +## Manual Structure + +The UTCP manual follows a standardized structure that defines your tools and how to call them. For complete field specifications, data types, and validation rules, see the [UTCP Manual API Reference](./api/core/utcp/data/utcp_manual.md). + +### Key Components + +- **Manual metadata**: Version information and API details +- **Tool definitions**: Description of available tools and their capabilities +- **Call templates**: Instructions for invoking each tool +- **Authentication**: Security configuration for tool access +- **Variables**: Dynamic values for tool parameters + +### Tool Definition + +Tools are defined in your UTCP manual with their input parameters, call instructions, and optional metadata. For complete field specifications, see the [Tool API Reference](./api/core/utcp/data/tool.md). + +## Communication Protocol Plugins + +### HTTP Tools + +Most common for REST APIs: + +```json +{ + "name": "create_user", + "description": "Create a new user account", + "inputs": { + "type": "object", + "properties": { + "user_fields": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "email": {"type": "string", "format": "email"} + }, + "required": ["name", "email"] + } + }, + "required": ["user_fields"] + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + }, + "body_field": "user_fields" + } +} +``` + +### CLI Tools + +For command-line applications: + +```json +{ + "name": "git_analysis", + "description": "Analyze git repository status and commit history", + "inputs": { + "type": "object", + "properties": { + "repo_path": {"type": "string", "description": "Path to git repository"} + }, + "required": ["repo_path"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_repo_path_UTCP_END", + "append_to_final_output": false + }, + { + "command": "git status --porcelain", + "append_to_final_output": false + }, + { + "command": "echo \"Status: $CMD_1_OUTPUT, Files changed: $(echo \"$CMD_1_OUTPUT\" | wc -l)\"", + "append_to_final_output": true + } + ] + } +} +``` + +## Authentication + +### API Key Authentication + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +### Bearer Token + +To use a Bearer Token, you can use the `api_key` authentication type. + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${ACCESS_TOKEN}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +### OAuth2 + +```json +{ + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "read:users write:users" + } +} +``` + +### Per-Tool Authentication + +```json +{ + "name": "admin_action", + "description": "Perform admin action", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/admin/action", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "${ADMIN_TOKEN}", + "var_name": "Authorization", + "location": "header" + } + } +} +``` + +## Variable Substitution + +Use `${VARIABLE_NAME}` syntax for dynamic values: + +### From Tool Arguments + +```json +{ + "url": "https://api.example.com/users/${user_id}", + "body": { + "name": "${name}", + "email": "${email}" + } +} +``` + +### From Environment Variables + +```json +{ + "headers": { + "Authorization": "Bearer ${API_TOKEN}", + "X-Client-ID": "${CLIENT_ID}" + } +} +``` + +### Default Values + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "variables": { + "base_url": "https://api.example.com", + "timeout": 30 + }, + "tools": [ + { + "name": "get_data", + "tool_call_template": { + "call_template_type": "http", + "url": "${base_url}/data", + "timeout": "${timeout}" + } + } + ] +} +``` + +## Implementation Examples + +### REST API Implementation + +For a typical REST API, you'll need to: + +1. **Keep your existing endpoints unchanged** +2. **Add a UTCP discovery endpoint** at `/utcp` +3. **Map your API operations to UTCP tools** + +Example API structure: +``` +GET /users/{user_id} # Your existing endpoint +POST /users # Your existing endpoint +GET /utcp # New UTCP discovery endpoint +``` + +The UTCP manual describes how to call your existing endpoints: + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "info": { + "title": "User Management API", + "version": "1.0.0", + "description": "API for managing user accounts" + }, + "tools": [ + { + "name": "get_user", + "description": "Retrieve user information by ID", + "inputs": { + "type": "object", + "properties": { + "user_id": {"type": "string"} + }, + "required": ["user_id"] + }, + "outputs": { + "type": "object", + "properties": { + "id": {"type": "string"}, + "name": {"type": "string"}, + "email": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users/${user_id}", + "http_method": "GET", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + } + } + }, + { + "name": "create_user", + "description": "Create a new user account", + "inputs": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "email": {"type": "string"} + }, + "required": ["name", "email"] + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + }, + "body_field": "user_data" + } + } + ] +} +``` + +## OpenAPI Integration + +If you already have an OpenAPI/Swagger specification, you can automatically convert it to a UTCP manual: + +### Automatic Conversion + +Many UTCP implementations provide OpenAPI converters that can: + +1. **Parse OpenAPI specifications** from URLs or files +2. **Convert paths to UTCP tools** automatically +3. **Map authentication schemes** to UTCP auth types +4. **Generate proper input/output schemas** + +### Conversion Configuration + +You can customize the conversion process: + +```json +{ + "source": "https://api.example.com/openapi.json", + "base_url": "https://api.example.com", + "include_operations": ["get", "post"], + "exclude_paths": ["/internal/*"], + "auth_template": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +### Manual Review + +After automatic conversion: +1. **Review generated tools** for accuracy +2. **Add missing descriptions** and examples +3. **Validate input/output schemas** +4. **Test with UTCP clients** +5. **Customize authentication** as needed + +## Best Practices + +### Manual Design + +1. **Clear Descriptions**: Write clear, concise tool descriptions +2. **Comprehensive Schemas**: Use detailed JSON schemas for inputs/outputs +3. **Consistent Naming**: Use consistent naming conventions +4. **Version Management**: Use semantic versioning for your manual +5. **Documentation**: Include examples and usage notes + +### Security + +1. **Authentication**: Always implement proper authentication +2. **Input Validation**: Validate all inputs on your API side +3. **Rate Limiting**: Implement rate limiting to prevent abuse +4. **HTTPS Only**: Use HTTPS for all production endpoints +5. **Credential Management**: Never hardcode credentials in manuals + +### Performance + +1. **Efficient Endpoints**: Design efficient API endpoints +2. **Caching**: Implement appropriate caching strategies +3. **Pagination**: Use pagination for large result sets +4. **Timeouts**: Set reasonable timeout values +5. **Monitoring**: Monitor API performance and usage + +### Maintenance + +1. **Versioning**: Version your manual and API together +2. **Backward Compatibility**: Maintain backward compatibility when possible +3. **Deprecation**: Provide clear deprecation notices +4. **Testing**: Test your manual with UTCP clients +5. **Documentation**: Keep documentation up to date + +## Testing Your Manual + +### Manual Validation + +Validate your UTCP manual structure: + +1. **JSON Schema Validation**: Ensure your manual follows the UTCP schema +2. **Tool Definition Validation**: Check that all tools have required fields +3. **Call Template Validation**: Verify call templates are properly formatted +4. **Authentication Validation**: Test authentication configurations + +### Integration Testing + +Test your manual with UTCP clients: + +1. **Tool Discovery**: Verify clients can discover your tools +2. **Tool Execution**: Test actual tool calls with various inputs +3. **Error Handling**: Test error scenarios and responses +4. **Authentication**: Verify authentication works correctly +5. **Performance**: Test response times and reliability + +### Testing Checklist + +- [ ] Manual validates against UTCP schema +- [ ] All tools have unique names +- [ ] All required fields are present +- [ ] Call templates are correctly formatted +- [ ] Authentication works as expected +- [ ] Tools return expected outputs +- [ ] Error responses are properly formatted +- [ ] Performance meets requirements + +## Migration Strategies + +### From OpenAPI + +1. **Automatic Conversion**: Use OpenAPI converter for initial conversion +2. **Manual Refinement**: Refine converted manual for better descriptions +3. **Authentication Setup**: Configure authentication properly +4. **Testing**: Test converted manual thoroughly + +### From MCP + +1. **Wrapper Approach**: Use UTCP-MCP plugin initially +2. **Gradual Migration**: Migrate tools one by one to native protocols +3. **Direct Implementation**: Implement tools using native UTCP protocols +4. **Deprecation**: Remove MCP dependency once migration is complete + +## Common Patterns + +### CRUD Operations + +```json +{ + "tools": [ + { + "name": "create_resource", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/resources", + "http_method": "POST" + } + }, + { + "name": "get_resource", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/resources/${id}", + "http_method": "GET" + } + }, + { + "name": "update_resource", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/resources/${id}", + "http_method": "PUT" + } + }, + { + "name": "delete_resource", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/resources/${id}", + "http_method": "DELETE" + } + } + ] +} +``` + +### Batch Operations + +```json +{ + "name": "batch_process", + "description": "Process multiple items in batch", + "inputs": { + "type": "object", + "properties": { + "items": { + "type": "array", + "items": {"type": "object"} + } + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/batch", + "http_method": "POST", + "body": { + "items": "${items}" + } + } +} +``` + +## Next Steps + +1. **Design Your Manual**: Plan your tool structure and descriptions +2. **Choose Protocols**: Select appropriate communication protocols +3. **Implement Discovery**: Add the `/utcp` endpoint to your API +4. **Test Integration**: Test with UTCP clients +5. **Monitor Usage**: Monitor how your tools are being used +6. **Iterate**: Improve based on usage patterns and feedback + +For more information, see: +- [Communication Protocol Plugins](./protocols/index.md) +- [Implementation Guide](./implementation.md) +- [Security Considerations](./security.md) + +## Language-Specific Implementation + +For detailed implementation examples and code samples in your programming language: + +- **Multi-language**: [UTCP Implementation Examples](https://github.com/universal-tool-calling-protocol) - Examples across Python, TypeScript, Go, and other languages +- **TypeScript**: [TypeScript UTCP Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/tree/main/docs) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) diff --git a/versioned_docs/version-1.0/implementation.md b/versioned_docs/version-1.0/implementation.md new file mode 100644 index 0000000..c1aee3a --- /dev/null +++ b/versioned_docs/version-1.0/implementation.md @@ -0,0 +1,396 @@ +--- +id: implementation +title: Implementation Guide +sidebar_position: 4 +--- + +# UTCP Implementation Guide + +This guide covers the core concepts and patterns for implementing UTCP in any programming language, whether you're building tool providers or tool consumers. + +## Quick Start + +### 1. Install UTCP Library + +Choose the UTCP implementation for your programming language: + +- **Python**: `pip install utcp utcp-http utcp-cli` +- **Node.js**: `npm install @utcp/core @utcp/http @utcp/cli` +- **Other languages**: Check the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) + +### 2. Create Your First Tool Provider + +Create an HTTP endpoint that serves a UTCP manual: + +**Endpoint**: `GET /utcp` +**Response**: +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "info": { + "title": "Weather API", + "version": "1.0.0" + }, + "tools": [ + { + "name": "get_weather", + "description": "Get current weather for a location", + "inputs": { + "type": "object", + "properties": { + "location": {"type": "string"} + }, + "required": ["location"] + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.weather.com/current", + "http_method": "GET", + "auth": { + "auth_type": "api_key", + "api_key": "${WEATHER_API_KEY}", + "var_name": "appid", + "location": "query" + } + } + } + ] +} +``` + +### 3. Create Your First Client + +Configure a UTCP client to discover and call tools: + +**Configuration**: +```json +{ + "manual_call_templates": [ + { + "name": "weather_service", + "call_template_type": "http", + "url": "https://api.weather.com/utcp", + "http_method": "GET" + } + ], + "variables": { + "WEATHER_API_KEY": "your-api-key" + } +} +``` + +**Usage**: +1. Initialize UTCP client with configuration +2. Discover tools from the weather service +3. Call the `get_weather` tool with location parameter + +## Core Concepts + +### UTCP Manual + +A UTCP manual is a JSON document that describes available tools and how to call them: + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "info": { + "title": "API Name", + "version": "1.0.0", + "description": "API description" + }, + "tools": [ + { + "name": "tool_name", + "description": "Tool description", + "inputs": { + "type": "object", + "properties": { + "param": {"type": "string"} + } + }, + "outputs": { + "type": "object", + "properties": { + "result": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "POST" + } + } + ] +} +``` + +### Call Templates + +Call templates define how to invoke tools using specific protocols: + +#### HTTP Call Template +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + }, + "body_field": "body" +} + +Tool arguments not used in the URL path or headers will be sent as query parameters for GET requests, or in the request body for POST/PUT/PATCH requests. The `body_field` specifies which tool argument contains the data for the request body. +} +``` + +#### CLI Call Template +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "cd /app", + "append_to_final_output": false + }, + { + "command": "python script.py UTCP_ARG_input_UTCP_END", + "append_to_final_output": true + } + ], + "env_vars": { + "PYTHONPATH": "/app/lib" + }, + "working_dir": "/app" +} +``` + +### Variable Substitution + +Variables in call templates are replaced with actual values: + +- **Tool arguments**: `${argument_name}` +- **Environment variables**: `${ENV_VAR}` +- **Configuration variables**: `${config.variable}` + +## Tool Provider Implementation + +### Manual Structure + +Create a well-structured UTCP manual: + +1. **Info Section**: Describe your API +2. **Tools Array**: Define each available tool +3. **Input Schemas**: Specify required parameters +4. **Output Schemas**: Document return values +5. **Call Templates**: Define how to invoke each tool + +### Discovery Endpoint + +Expose your manual via HTTP: + +``` +GET /utcp +Content-Type: application/json + +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [...] +} +``` + +### Authentication + +Support various authentication methods: + +#### API Key Authentication +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +#### OAuth2 Authentication +```json +{ + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token" + } +} +``` + +## Tool Consumer Implementation + +### Client Configuration + +Configure your UTCP client: + +#### File-based Configuration +```yaml +# utcp-config.yaml +manual_call_templates: + - name: weather_service + call_template_type: http + url: https://api.weather.com/utcp + +variables: + WEATHER_API_KEY: your-api-key + +load_variables_from: + - variable_loader_type: dotenv + env_file_path: .env +``` + +#### Programmatic Configuration +```json +{ + "manual_call_templates": [ + { + "name": "service_name", + "call_template_type": "http", + "url": "https://api.example.com/utcp" + } + ], + "variables": { + "API_KEY": "your-key" + } +} +``` + +### Tool Discovery + +Discover available tools: + +1. **List Tools**: Get all available tools +2. **Tool Information**: Get detailed tool metadata +3. **Filter Tools**: Find tools by name, tags, or description + +### Tool Execution + +Execute tools with proper error handling: + +1. **Basic Calls**: Simple tool invocation +2. **Batch Calls**: Execute multiple tools +3. **Context Passing**: Pass context between calls +4. **Error Handling**: Handle failures gracefully + +## Advanced Implementation Patterns + +### Custom Protocol Plugins + +Extend UTCP with custom communication protocols: + +1. **Define Call Template**: Structure for your protocol +2. **Implement Communication Handler**: Protocol-specific logic +3. **Register Protocol**: Make it available to clients + +Example custom protocol structure: +```json +{ + "call_template_type": "custom", + "custom_field": "value", + "timeout": 30 +} +``` + +### Custom Tool Repositories + +Implement custom tool storage: + +1. **Tool Storage**: How tools are stored and retrieved +2. **Search Functionality**: Tool discovery and filtering +3. **Caching**: Performance optimization +4. **Synchronization**: Multi-client coordination + +### Testing Strategies + +#### Unit Testing Tool Providers +- Test manual generation +- Validate tool definitions +- Test authentication +- Mock external dependencies + +#### Integration Testing +- Test tool discovery +- Test tool execution +- Test error scenarios +- Test performance + +#### End-to-End Testing +- Test complete workflows +- Test multiple protocols +- Test real-world scenarios +- Test scalability + +## Best Practices + +### Performance +- Use connection pooling +- Implement caching +- Optimize tool discovery +- Monitor response times + +### Security +- Validate all inputs +- Use secure authentication +- Implement rate limiting +- Log security events + +### Reliability +- Implement retry logic +- Handle network failures +- Use circuit breakers +- Monitor tool health + +### Maintainability +- Version your manuals +- Document all tools +- Use consistent naming +- Provide examples + +## Deployment Considerations + +### Scaling +- Load balance tool providers +- Cache tool discoveries +- Use async processing +- Monitor resource usage + +### Monitoring +- Track tool usage +- Monitor error rates +- Log performance metrics +- Set up alerts + +### Security +- Use HTTPS everywhere +- Implement proper authentication +- Validate all inputs +- Monitor for abuse + +## Language-Specific Implementation + +For detailed implementation examples and code samples in your programming language: + +- **Multi-language**: [UTCP Implementation Examples](https://github.com/universal-tool-calling-protocol) - Examples across Python, TypeScript, Go, and other languages +- **TypeScript**: [TypeScript UTCP Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/tree/main/docs) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) + +For more detailed information, see: +- [Communication Protocol Plugins](./protocols/index.md) +- [API Reference](./api/index.md) +- [Security Considerations](./security.md) diff --git a/versioned_docs/version-1.0/index.md b/versioned_docs/version-1.0/index.md index 4f0a4e4..2355a3e 100644 --- a/versioned_docs/version-1.0/index.md +++ b/versioned_docs/version-1.0/index.md @@ -4,152 +4,219 @@ title: Introduction sidebar_position: 1 --- -# Introduction to UTCP 1.0 +# Universal Tool Calling Protocol (UTCP) -The Universal Tool Calling Protocol (UTCP) is a lightweight, secure, and scalable standard for defining and interacting with tools across a wide variety of communication protocols. Version 1.0 introduces a modular core with a plugin-based architecture, making it more extensible, testable, and easier to package. - -## Core Components +:::info Language Examples +UTCP is available in multiple languages - see [Python](https://github.com/universal-tool-calling-protocol/python-utcp), [TypeScript](https://github.com/universal-tool-calling-protocol/typescript-utcp), [Go](https://github.com/universal-tool-calling-protocol/go-utcp), and other implementations in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol). +::: -UTCP consists of four main components: +UTCP is a lightweight, secure, and scalable standard that enables AI agents and applications to discover and call tools directly using their native protocols - **no wrapper servers required**. -1. [**Manuals**](./api/core/utcp/data/utcp_manual.md): The standard tool provider description format that contains tool definitions -2. [**Tools**](./api/core/utcp/data/tool.md): The individual capabilities that can be called -3. [**Call Templates**](./api/core/utcp/data/call_template.md): The communication configurations that specify how tools are accessed. Concretely this maps a tool name and provided arguments to an actual API request in a communication protocol. -4. [**UtcpClient**](./api/core/utcp/utcp_client.md): The client that calls tools using the call templates. +## Why UTCP? -## The "Manual" Approach +### The Problem with Current Approaches +Most tool integration solutions force you to: +- Build and maintain wrapper servers for every tool +- Route all traffic through a middleman protocol +- Reimplement existing authentication and security +- Accept additional latency and complexity -UTCP's fundamental philosophy is to act as a descriptive manual rather than a prescriptive middleman: +### The UTCP Solution +UTCP acts as a **"manual"** that tells agents how to call your tools directly: -:::note -A UTCP Manual tells an agent: "Here is a tool. Here is its native endpoint (HTTP, WebSocket, CLI, etc.), and here is how to call it directly." +:::tip Core Philosophy +*"If a human can call your API, an AI agent should be able to call it too - with the same security and no additional infrastructure."* ::: -This approach eliminates the need for wrapper servers and allows direct communication between agents and tools. - -## New Architecture in 1.0 - -UTCP has been refactored into a core library and a set of optional plugins: - -### Core Package (`utcp`) -- **Data Models**: Pydantic models for [`Tool`](./api/core/utcp/data/tool.md), [`CallTemplate`](./api/core/utcp/data/call_template.md), [`UtcpManual`](./api/core/utcp/data/utcp_manual.md), and [`Auth`](./api/core/utcp/data/auth.md) -- **Pluggable Interfaces**: [`CommunicationProtocol`](./api/core/utcp/interfaces/communication_protocol.md), [`ConcurrentToolRepository`](./api/core/utcp/interfaces/concurrent_tool_repository.md), [`ToolSearchStrategy`](./api/core/utcp/interfaces/tool_search_strategy.md), [`VariableSubstitutor`](./api/core/utcp/interfaces/variable_substitutor.md), [`ToolPostProcessor`](./api/core/utcp/interfaces/tool_post_processor.md) -- **Default Implementations**: [`UtcpClient`](./api/core/utcp/utcp_client.md), [`InMemToolRepository`](./api/core/utcp/implementations/in_mem_tool_repository.md), [`TagAndDescriptionWordMatchStrategy`](./api/core/utcp/implementations/tag_search.md) - -### Protocol Plugins -- `utcp-http`: Supports HTTP, SSE, and streamable HTTP, plus an OpenAPI converter -- `utcp-cli`: For wrapping local command-line tools -- `utcp-mcp`: For interoperability with the Model Context Protocol (MCP) -- `utcp-text`: For reading text files -- `utcp-socket`: TCP and UDP protocols (work in progress) -- `utcp-gql`: GraphQL (work in progress) - -## Minimal Example - -Let's see how easy it is to use UTCP with a minimal example. - -### 1. Defining a Tool (Tool Provider) - -Create a simple HTTP endpoint that serves a UTCP Manual (JSON): - -```python -# app.py -from fastapi import FastAPI - -app = FastAPI() - -@app.get("/utcp") -def utcp_discovery(): - return { - "manual_version": "1.0.0", - "utcp_version": "1.0.1", - "tools": [ - { - "name": "get_weather", - "description": "Get current weather for a location", - "inputs": { - "type": "object", - "properties": { - "location": { - "type": "string", - "description": "City name" - } - }, - "required": ["location"] - }, - "outputs": { - "type": "object", - "properties": { - "temperature": {"type": "number"}, - "conditions": {"type": "string"} - } - }, - "tool_call_template": { - "call_template_type": "http", - "url": "https://example.com/api/weather", - "http_method": "GET" - } - } - ] - } +## Quick Start (5 Minutes) + +### 1. Install UTCP + +```bash +# Example installation (Python) +pip install utcp utcp-http + +# Example installation (Node.js) +npm install @utcp/core @utcp/http -# Implement the actual weather API endpoint -@app.get("/api/weather") -def get_weather(location: str): - # In a real app, you'd fetch actual weather data - return {"temperature": 22.5, "conditions": "Sunny"} +# See language-specific documentation for other implementations ``` -Run the server: +### 2. Expose Your First Tool + +Add a discovery endpoint to your existing API: + +**Add endpoint**: `GET /utcp` +**Return your UTCP manual**: +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [{ + "name": "get_weather", + "description": "Get current weather for a location", + "inputs": { + "type": "object", + "properties": {"location": {"type": "string"}}, + "required": ["location"] + }, + "tool_call_template": { + "call_template_type": "http", + "url": "http://localhost:8000/weather", + "http_method": "GET" + } + }], + "auth": { + "auth_type": "api_key", + "api_key": "${WEATHER_API_KEY}", + "var_name": "appid", + "location": "query" + } + } + }] +} +``` -```bash -uvicorn app:app --reload +### 3. Call Your Tool + +**Configure UTCP client**: +```json +{ + "manual_call_templates": [{ + "name": "weather_api", + "call_template_type": "http", + "url": "http://localhost:8000/utcp", + "http_method": "GET" + }] +} ``` -### 2. Using the Tool (Client) - -```python -# client.py -import asyncio -from utcp.utcp_client import UtcpClient -from utcp_http.http_call_template import HttpCallTemplate - -async def main(): - # Create a UTCP client with configuration - client = await UtcpClient.create(config={ - "manual_call_templates": [ - { - "name": "weather_service", - "call_template_type": "http", - "http_method": "GET", - "url": "http://localhost:8000/utcp" - } - ] - }) - - # Tools are automatically registered from the manual call templates - # Call a tool by its namespaced name: {manual_name}.{tool_name} - result = await client.call_tool( - "weather_service.get_weather", - tool_args={"location": "San Francisco"} - ) - print(f"Weather: {result['temperature']}°C, {result['conditions']}") - -if __name__ == "__main__": - asyncio.run(main()) +**Call the tool**: +1. Initialize UTCP client with configuration +2. Discover tools from weather API +3. Call `get_weather` tool with location parameter +4. Receive weather data response ``` -Run the client: +**That's it!** Your tool is now discoverable and callable by any UTCP client. -```bash -python client.py +## Key Benefits + +| Benefit | Description | +|---------|-------------| +| **🚀 Zero Latency Overhead** | Direct tool calls, no proxy servers | +| **🔒 Native Security** | Use your existing authentication and authorization | +| **🌐 Protocol Flexibility** | HTTP, WebSocket, CLI, GraphQL, and more | +| **⚡ Easy Integration** | Add one endpoint, no infrastructure changes | +| **📈 Scalable** | Leverage your existing scaling and monitoring | + +## How It Works + +```mermaid +graph LR + A[AI Agent] -->|1. Discover| B[UTCP Manual] + B -->|2. Learn| C[Tool Definitions] + A -->|3. Call Directly| D[Your API] + D -->|4. Response| A ``` -## Benefits of the UTCP Approach +1. **Discovery**: Agent fetches your UTCP manual +2. **Learning**: Agent understands how to call your tools +3. **Direct Calling**: Agent calls your API directly using native protocols +4. **Response**: Your API responds normally + +## Supported Protocols + +UTCP supports multiple communication protocols through plugins: + +| Protocol | Use Case | Plugin | Status | +|----------|----------|--------|--------| +| **[HTTP](./protocols/http.md)** | REST APIs, webhooks | `utcp-http` | ✅ Stable | +| **[WebSocket](./protocols/websocket.md)** | Real-time communication | `utcp-websocket` | ✅ Stable | +| **[CLI](./protocols/cli.md)** | Command-line tools | `utcp-cli` | ✅ Stable | +| **[Server-Sent Events](./protocols/streamable-http.md)** | Streaming data | `utcp-http` | ✅ Stable | +| **[Text Files](./protocols/text.md)** | File reading | `utcp-text` | ✅ Stable | +| **[MCP](./protocols/mcp.md)** | MCP interoperability | `utcp-mcp` | ✅ Stable | + +[View all protocols →](./protocols/index.md) + +## Architecture Overview -1. **Direct Communication**: The client calls the tool's native endpoint directly -2. **No Wrapper Infrastructure**: No need to build and maintain wrapper servers -3. **Leverage Existing Systems**: Uses the tool's existing authentication, rate limiting, etc. -4. **Flexible Protocol Support**: Works with any communication protocol (HTTP, WebSockets, CLI, etc.) +UTCP v1.0 features a modular, plugin-based architecture: + +### Core Components +- **[Manuals](./api/core/utcp/data/utcp_manual.md)**: Tool definitions and metadata +- **[Tools](./api/core/utcp/data/tool.md)**: Individual callable capabilities +- **[Call Templates](./api/core/utcp/data/call_template.md)**: Protocol-specific call instructions +- **[UTCP Client](./api/core/utcp/utcp_client.md)**: Tool discovery and execution engine + +### Plugin System +- **Protocol Plugins**: HTTP, WebSocket, CLI, etc. +- **Custom Protocols**: Extend with your own communication methods +- **Tool Repositories**: Pluggable storage for tool definitions +- **Search Strategies**: Customizable tool discovery algorithms + +[Learn more about the architecture →](./api/index.md) + +## Who Should Use UTCP? + +### 🛠️ Tool Providers +You have APIs, services, or tools that you want AI agents to use: +- **Existing API owners** - Expose your REST APIs to AI agents +- **SaaS providers** - Make your services AI-accessible +- **Enterprise teams** - Enable internal tool usage by AI systems + +[**Get started as a tool provider →**](./for-tool-providers.md) + +### 🤖 Tool Consumers +You're building AI agents or applications that need to call external tools: +- **AI agent developers** - Give your agents access to external capabilities +- **Application builders** - Integrate third-party tools seamlessly +- **Enterprise developers** - Connect to internal and external services + +[**Get started as a tool consumer →**](./implementation.md) + +## UTCP vs Alternatives + +| Feature | UTCP | MCP | Custom Wrappers | +|---------|------|-----|-----------------| +| **Infrastructure** | None required | Wrapper servers | Custom servers | +| **Latency** | Direct calls | Double hop | Variable | +| **Security** | Native | Reimplemented | Custom | +| **Protocols** | Multiple | HTTP streaming | Single | +| **Maintenance** | Minimal | High | Very high | + +[**Detailed comparison with MCP →**](./utcp-vs-mcp.md) + +## Next Steps + +### For Tool Providers +1. **[Read the provider guide](./for-tool-providers.md)** - Learn how to expose your tools +2. **[Choose your protocol](./protocols/index.md)** - Select the right communication method +3. **[Implement your manual](./implementation.md)** - Add UTCP to your existing API +4. **[Secure your tools](./security.md)** - Implement proper authentication + +### For Tool Consumers +1. **[Read the implementation guide](./implementation.md)** - Learn how to build UTCP clients +2. **[Explore protocols](./protocols/index.md)** - Understand available communication options +3. **[Check examples](https://github.com/universal-tool-calling-protocol)** - See real-world implementations +4. **[Join the community](https://discord.gg/ZpMbQ8jRbD)** - Get help and share experiences + +### Migration from Other Systems +- **[From UTCP v0.1](./migration-v0.1-to-v1.0.md)** - Upgrade to the latest version +- **[From MCP](./protocols/mcp.md)** - Migrate from Model Context Protocol +- **[From custom solutions](./implementation.md)** - Replace existing tool integrations + +## Community & Support + +- **[GitHub Organization](https://github.com/universal-tool-calling-protocol)** - Source code and issues +- **[Discord Community](https://discord.gg/ZpMbQ8jRbD)** - Real-time help and discussions +- **[Tool Registry](https://utcp.io/registry)** - Discover available tools +- **[RFC Process](/about/RFC)** - Contribute to the specification + +--- -In the following sections, we'll explore the details of UTCP's components and how to implement them in your applications. +**Ready to get started?** Choose your path: +- 🛠️ [**I want to expose my tools**](./for-tool-providers.md) +- 🤖 [**I want to call tools**](./implementation.md) +- 📚 [**I want to learn more**](./protocols/index.md) diff --git a/versioned_docs/version-1.0/migration-v0.1-to-v1.0.md b/versioned_docs/version-1.0/migration-v0.1-to-v1.0.md new file mode 100644 index 0000000..eb50de7 --- /dev/null +++ b/versioned_docs/version-1.0/migration-v0.1-to-v1.0.md @@ -0,0 +1,465 @@ +--- +id: migration-v0.1-to-v1.0 +title: Migration Guide - v0.1 to v1.0 +sidebar_position: 7 +--- + +# Migration Guide: v0.1 to v1.0 + +This guide helps you migrate from UTCP v0.1 to v1.0, which introduces significant architectural improvements and new features. + +## Overview of Changes + +### Major Changes in v1.0 + +1. **Plugin Architecture**: Core functionality split into pluggable components +2. **Improved Data Models**: Enhanced Pydantic models with better validation +3. **New Protocol Support**: Additional communication protocols +4. **Better Error Handling**: More specific exception types +5. **Enhanced Authentication**: Expanded authentication options +6. **Performance Improvements**: Optimized client and protocol implementations + +### Breaking Changes + +| Component | v0.1 | v1.0 | Impact | +|-----------|------|------|--------| +| **Package Structure** | Single package | Core + plugins | High | +| **Client API** | `UtcpClient()` | `UtcpClient.create()` | Medium | +| **Configuration** | Simple dict | Structured config | Medium | +| **Protocol Registration** | Automatic | Plugin-based | High | +| **Error Types** | Generic exceptions | Specific exception types | Low | + +## Installation Changes + +### v0.1 Installation + +```bash +pip install utcp +``` + +### v1.0 Installation + +```bash +# Core package +pip install utcp + +# Protocol plugins (install as needed) +pip install utcp-http utcp-cli utcp-websocket utcp-text utcp-mcp +``` + +## Client Migration + +### v0.1 Client Code + +**Legacy v0.1 client configuration:** +- Import UTCP client library +- Configure providers with provider-specific settings (name, type, URL, HTTP method) +- Call tools using provider.tool_name format +- Synchronous tool calling interface + +### v1.0 Client Code + +**New v1.0 client configuration:** +- Import updated UTCP client library from new module path +- Use async factory method for client creation +- Configure manual call templates instead of providers +- Use async/await pattern for tool calling +- Enhanced error handling and response processing + +## Configuration Migration + +### v0.1 Configuration + +```yaml +providers: + - name: weather_service + provider_type: http + url: https://weather.example.com/utcp + http_method: GET + - name: file_service + provider_type: cli + command: cat + args: ["${filename}"] + +variables: + API_KEY: your_api_key +``` + +### v1.0 Configuration + +```yaml +manual_call_templates: + - name: weather_service + call_template_type: http + url: https://weather.example.com/utcp + http_method: GET + - name: file_service + call_template_type: cli + command: cat + args: ["${filename}"] + +load_variables_from: + - variable_loader_type: dotenv + env_file_path: .env +``` + +## Manual Format Migration + +### v0.1 Manual Format + +```json +{ + "utcp_version": "0.1.0", + "provider_info": { + "name": "weather_api", + "version": "1.0.0" + }, + "tools": [ + { + "name": "get_weather", + "description": "Get weather data", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"} + } + }, + "provider": { + "provider_type": "http", + "url": "https://api.weather.com/current", + "method": "GET" + } + } + ] +} +``` + +### v1.0 Manual Format + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "info": { + "title": "Weather API", + "version": "1.0.0", + "description": "Weather data and forecasting tools" + }, + "tools": [ + { + "name": "get_weather", + "description": "Get weather data", + "inputs": { + "type": "object", + "properties": { + "location": {"type": "string"} + }, + "required": ["location"] + }, + "outputs": { + "type": "object", + "properties": { + "temperature": {"type": "number"}, + "conditions": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.weather.com/current", + "http_method": "GET", + }, + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "appid", + "location": "query" + } + } + } + ] +} +``` + +## Protocol Migration + +### HTTP Protocol Changes + +#### v0.1 HTTP Provider + +```json +{ + "provider_type": "http", + "url": "https://api.example.com/endpoint", + "method": "POST", + "headers": {"Authorization": "Bearer ${TOKEN}"}, + "body": {"data": "${input}"} +} +``` + +#### v1.0 HTTP Call Template + +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "POST", + "headers": {"Authorization": "Bearer ${TOKEN}"}, + "body_field": "body", + "auth": { + "auth_type": "api_key", + "api_key": "${TOKEN}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +### CLI Protocol Changes + +#### v0.1 CLI Provider + +```json +{ + "provider_type": "cli", + "command": "python", + "args": ["script.py", "${input}"], + "cwd": "/app" +} +``` + +#### v1.0 CLI Call Template + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "cd /app", + "append_to_final_output": false + }, + { + "command": "python script.py UTCP_ARG_input_UTCP_END", + "append_to_final_output": true + } + ], + "working_dir": "/app", + "env_vars": { + "PYTHONPATH": "/app/lib" + } +} +``` + +## Authentication Migration + +### v0.1 Authentication + +```json +{ + "provider": { + "provider_type": "http", + "url": "https://api.example.com/data", + "headers": { + "Authorization": "Bearer ${API_TOKEN}" + } + } +} +``` + +### v1.0 Authentication + +```json +{ + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/data", + "auth": { + "auth_type": "api_key", + "api_key": "${API_TOKEN}", + "var_name": "Authorization", + "location": "header" + } + } +} +``` + +## Error Handling Migration + +### v0.1 Error Handling + +**Basic error handling in v0.1:** +- Simple try/catch block with generic Exception handling +- Limited error information and context +- Basic error message printing + +### v1.0 Error Handling + +**Enhanced error handling in v1.0:** +- Import specific exception types from utcp.exceptions module +- Handle ToolNotFoundError for missing tools +- Handle AuthenticationError for auth failures +- Handle ToolCallError for tool execution failures +- Handle base UtcpError for general UTCP errors +- Use try/catch blocks with specific exception handling + +## Step-by-Step Migration + +### Step 1: Update Dependencies + +```bash +# Uninstall old version +pip uninstall utcp + +# Install new version with plugins +pip install utcp utcp-http utcp-cli utcp-websocket utcp-text +``` + +### Step 2: Update Client Code + +**Migration to async pattern:** +- **Before**: Synchronous client creation and tool calls +- **After**: Async client factory method and async tool calls +- Import asyncio module for async execution +- Use async/await keywords for client creation and tool calls +- Run async main function with asyncio.run() + +### Step 3: Update Configuration + +**Configuration migration helper:** +- Create migration function to convert v0.1 config to v1.0 format +- Transform providers array to manual_call_templates array +- Add variable_loaders configuration for environment variables +- Map provider_type to call_template_type +- Migrate HTTP provider settings (URL, method, headers, body) +- Migrate CLI provider settings (command, args, working_directory) +- Load old configuration and apply migration helper +- Create new client with migrated configuration + +### Step 4: Update Manual Format + +**Manual migration helper:** +- Create migration function to convert v0.1 manual to v1.0 format +- Update manual_version and utcp_version fields +- Transform provider_info to info structure (title, version, description) +- Migrate tools array with updated structure +- Convert tool parameters to inputs field +- Transform provider configuration to tool_call_template +- Map provider_type to call_template_type +- Migrate HTTP provider settings (URL, method, headers, body) + +### Step 5: Test Migration + +**Testing migrated client:** +- Import pytest and UtcpClient for async testing +- Create test function with pytest.mark.asyncio decorator +- Test client creation with migrated configuration +- Test tool discovery functionality (list_tools) +- Test tool calling with sample parameters +- Assert expected results and functionality + +## Common Migration Issues + +### Issue 1: Async/Await + +**Problem**: v1.0 client methods are async +**Solution**: Add `async`/`await` keywords + +**Code changes:** +- **Before**: Synchronous tool calling (result = client.call_tool("tool", args)) +- **After**: Async tool calling (result = await client.call_tool("tool", args)) + +### Issue 2: Configuration Format + +**Problem**: Configuration structure changed +**Solution**: Use migration helper or update manually + +**Configuration changes:** +- **Before**: Use "providers" array in configuration +- **After**: Use "manual_call_templates" array in configuration + +### Issue 3: Plugin Dependencies + +**Problem**: Protocol implementations not found +**Solution**: Install required plugins + +```bash +pip install utcp-http utcp-cli utcp-websocket +``` + +### Issue 4: Manual Format + +**Problem**: Old manual format not recognized +**Solution**: Update manual structure + +```json +// Before +{"provider": {"provider_type": "http"}} + +// After +{"tool_call_template": {"call_template_type": "http"}} +``` + +## Validation Tools + +### Configuration Validator + +**Configuration validation helper:** +- Import UtcpClientConfig from utcp.data.utcp_client_config +- Create validation function that accepts configuration dictionary +- Use UtcpClientConfig constructor to validate configuration structure +- Handle validation exceptions and provide error messages +- Return validated config object or None on failure + +### Manual Validator + +**Manual validation helper:** +- Import UtcpManual from utcp.data.utcp_manual +- Create validation function that accepts manual dictionary +- Use UtcpManual constructor to validate manual structure +- Handle validation exceptions and provide error messages +- Return validated manual object or None on failure + +## Best Practices for Migration + +1. **Gradual Migration**: Migrate one component at a time +2. **Test Thoroughly**: Test each migrated component +3. **Backup Configurations**: Keep backups of v0.1 configurations +4. **Use Validation**: Validate configurations and manuals +5. **Monitor Performance**: Check for performance regressions +6. **Update Documentation**: Update internal documentation +7. **Train Team**: Ensure team understands new patterns + +## Post-Migration Checklist + +- [ ] All dependencies updated +- [ ] Client code uses async/await +- [ ] Configuration format updated +- [ ] Manual format updated +- [ ] Error handling updated +- [ ] Tests passing +- [ ] Performance acceptable +- [ ] Documentation updated +- [ ] Team trained on changes + +## Getting Help + +If you encounter issues during migration: + +1. **Check Documentation**: Review the [Implementation Guide](./implementation.md) +2. **GitHub Issues**: Search existing issues or create new ones +3. **Discord Community**: Join the [UTCP Discord](https://discord.gg/ZpMbQ8jRbD) +4. **Examples**: Check the [examples repository](https://github.com/universal-tool-calling-protocol) for implementations across multiple languages + +## Rollback Plan + +If migration issues occur, you can rollback: + +```bash +# Rollback to v0.1 +pip uninstall utcp utcp-http utcp-cli utcp-websocket utcp-text +pip install utcp==0.1.0 + +# Restore old configuration files +cp config-v0.1-backup.yaml config.yaml +``` + +Remember to test the rollback process in a non-production environment first. diff --git a/versioned_docs/version-1.0/protocols/cli.md b/versioned_docs/version-1.0/protocols/cli.md new file mode 100644 index 0000000..f0ac213 --- /dev/null +++ b/versioned_docs/version-1.0/protocols/cli.md @@ -0,0 +1,471 @@ +--- +id: cli +title: CLI Protocol +sidebar_position: 4 +--- + +# CLI Protocol + +The CLI protocol plugin (`utcp-cli`) enables UTCP to execute multi-command workflows and scripts. This protocol supports sequential command execution with state preservation and cross-platform compatibility. + +## Installation + +```bash +# Example installation (Python) +pip install utcp-cli + +# Example installation (Node.js) +npm install @utcp/cli +``` + +## Key Features + +- **Multi-Command Execution**: Execute multiple commands sequentially in a single subprocess +- **State Preservation**: Directory changes and environment persist between commands +- **Cross-Platform Script Generation**: PowerShell on Windows, Bash on Unix/Linux/macOS +- **Flexible Output Control**: Choose which command outputs to include in final result +- **Argument Substitution**: `UTCP_ARG_argname_UTCP_END` placeholder system +- **Output Referencing**: Access previous command outputs with `$CMD_0_OUTPUT`, `$CMD_1_OUTPUT` +- **Environment Variables**: Secure credential and configuration passing + +## Call Template Structure + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_target_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "git status --porcelain", + "append_to_final_output": false + }, + { + "command": "echo \"Status: $CMD_1_OUTPUT, Files: $(echo \"$CMD_1_OUTPUT\" | wc -l)\"", + "append_to_final_output": true + } + ], + "working_dir": "/tmp", + "env_vars": { + "GIT_AUTHOR_NAME": "UTCP Bot" + } +} +``` + +## Configuration Options + +### Required Fields + +- **`call_template_type`**: Must be `"cli"` +- **`commands`**: Array of `CommandStep` objects defining the sequence of commands + +### Optional Fields + +- **`working_dir`**: Working directory for command execution +- **`env_vars`**: Environment variables to set for all commands + +### CommandStep Object + +- **`command`**: Command string with `UTCP_ARG_argname_UTCP_END` placeholders +- **`append_to_final_output`**: Whether to include this command's output in the final result (defaults to `false` for all except the last command) + +For complete field specifications and validation rules, see the [CLI Call Template API Reference](../api/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.md). + +## Argument Substitution + +Use `UTCP_ARG_argname_UTCP_END` placeholders in command strings: + +```json +{ + "command": "git clone UTCP_ARG_repo_url_UTCP_END UTCP_ARG_target_dir_UTCP_END" +} +``` + +## Output Referencing + +Reference previous command outputs using `$CMD_N_OUTPUT` variables: + +```json +{ + "commands": [ + { + "command": "git status --porcelain", + "append_to_final_output": false + }, + { + "command": "echo \"Changes detected: $CMD_0_OUTPUT\"", + "append_to_final_output": true + } + ] +} +``` + +## Security Considerations + +:::danger Security Warning +CLI protocol executes commands on the local system. Always validate inputs and use with caution. +::: + +### Input Validation + +Always validate and sanitize inputs to prevent command injection: + +```json +{ + "name": "safe_file_read", + "description": "Safely read a file", + "inputs": { + "type": "object", + "properties": { + "filename": { + "type": "string", + "pattern": "^[a-zA-Z0-9._-]+$" + } + }, + "required": ["filename"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "cd /safe/directory", + "append_to_final_output": false + }, + { + "command": "cat UTCP_ARG_filename_UTCP_END", + "append_to_final_output": true + } + ], + "working_dir": "/safe/directory" + } +} +``` + +### Sandboxing + +Consider running CLI tools in sandboxed environments: + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "docker run --rm --read-only -v /safe/data:/data:ro alpine:latest cat /data/UTCP_ARG_filename_UTCP_END", + "append_to_final_output": true + } + ] +} +``` + +## Examples + +### Simple Command Execution + +```json +{ + "name": "get_system_info", + "description": "Get system information", + "inputs": { + "type": "object", + "properties": {} + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "uname -a", + "append_to_final_output": true + } + ] + } +} +``` + +### Multi-Step File Analysis + +```json +{ + "name": "analyze_directory", + "description": "Analyze files in a directory", + "inputs": { + "type": "object", + "properties": { + "path": { + "type": "string", + "description": "Directory path to analyze" + } + }, + "required": ["path"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_path_UTCP_END", + "append_to_final_output": false + }, + { + "command": "find . -type f | wc -l", + "append_to_final_output": false + }, + { + "command": "du -sh .", + "append_to_final_output": false + }, + { + "command": "echo \"Directory Analysis: $CMD_2_OUTPUT total size, $CMD_1_OUTPUT files\"", + "append_to_final_output": true + } + ] + } +} +``` + +### Git Workflow + +```json +{ + "name": "git_analysis", + "description": "Analyze git repository", + "inputs": { + "type": "object", + "properties": { + "repo_url": {"type": "string"}, + "target_dir": {"type": "string"} + }, + "required": ["repo_url", "target_dir"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "git clone UTCP_ARG_repo_url_UTCP_END UTCP_ARG_target_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "cd UTCP_ARG_target_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "git log --oneline -10", + "append_to_final_output": true + }, + { + "command": "find . -name '*.py' | wc -l", + "append_to_final_output": false + }, + { + "command": "echo \"Repository has $CMD_3_OUTPUT Python files\"", + "append_to_final_output": true + } + ], + "env_vars": { + "GIT_AUTHOR_NAME": "UTCP Bot", + "GIT_AUTHOR_EMAIL": "bot@utcp.dev" + } + } +} +``` + +### Python Development Pipeline + +```json +{ + "name": "python_pipeline", + "description": "Run Python development pipeline", + "inputs": { + "type": "object", + "properties": { + "project_dir": {"type": "string"}, + "script_name": {"type": "string"} + }, + "required": ["project_dir", "script_name"] + }, + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "cd UTCP_ARG_project_dir_UTCP_END", + "append_to_final_output": false + }, + { + "command": "python -m pip install -r requirements.txt", + "append_to_final_output": false + }, + { + "command": "python UTCP_ARG_script_name_UTCP_END", + "append_to_final_output": true + } + ], + "working_dir": "/workspace", + "env_vars": { + "PYTHONPATH": "/workspace/lib", + "VIRTUAL_ENV": "/workspace/venv" + } + } +} +``` + +## Output Handling + +The CLI protocol executes all commands in a single subprocess and returns the combined output based on `append_to_final_output` settings: + +- Commands with `append_to_final_output: true` contribute to the final result +- Commands with `append_to_final_output: false` are executed for side effects only +- If no `append_to_final_output` is specified, only the last command's output is returned +- JSON output is automatically parsed if detected + +### Success Response + +```json +"Directory Analysis: 2.1G total size, 1247 files" +``` + +### JSON Output Detection + +If output starts with `{` or `[`, it's automatically parsed as JSON: + +```json +{ + "files": 1247, + "size": "2.1G", + "analysis_time": "2023-12-01T10:30:00Z" +} +``` + +## Environment Variables + +Set environment variables for all commands: + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "node app.js", + "append_to_final_output": true + } + ], + "env_vars": { + "NODE_ENV": "production", + "API_KEY": "${API_KEY}", + "PORT": "3000" + } +} +``` + +## Working Directory + +Specify the initial working directory: + +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "make build", + "append_to_final_output": true + } + ], + "working_dir": "/project/src" +} +``` + +## Cross-Platform Considerations + +### Command Syntax + +Commands should use appropriate syntax for the target platform: + +**Windows (PowerShell):** +```json +{ + "commands": [ + {"command": "Get-ChildItem UTCP_ARG_path_UTCP_END"}, + {"command": "Set-Location UTCP_ARG_dir_UTCP_END"} + ] +} +``` + +**Unix/Linux/macOS (Bash):** +```json +{ + "commands": [ + {"command": "ls -la UTCP_ARG_path_UTCP_END"}, + {"command": "cd UTCP_ARG_dir_UTCP_END"} + ] +} +``` + +### Universal Commands + +Some commands work across platforms: +```json +{ + "commands": [ + {"command": "git status"}, + {"command": "python --version"}, + {"command": "node -v"} + ] +} +``` + +## Best Practices + +1. **Validate Inputs**: Always validate and sanitize user inputs using JSON Schema patterns +2. **Use Absolute Paths**: Prefer absolute paths for commands and files when possible +3. **Control Output**: Use `append_to_final_output` to control which command outputs are returned +4. **Reference Previous Output**: Use `$CMD_N_OUTPUT` to reference previous command results +5. **Limit Permissions**: Run with minimal necessary permissions +6. **Sandbox Execution**: Use containers or chroot when possible +7. **Handle Cross-Platform**: Consider platform-specific command syntax +8. **Environment Variables**: Use `env_vars` for secure credential passing + +## Common Use Cases + +- **Multi-step Builds**: setup → compile → test → package +- **Git Workflows**: clone → analyze → commit → push +- **Data Pipelines**: fetch → transform → validate → output +- **File Operations**: navigate → search → process → report +- **Development Tools**: install dependencies → run tests → generate docs +- **System Administration**: check status → backup → cleanup → verify + +## Error Handling + +| Error Type | Description | Handling | +|------------|-------------|---------| +| Missing Arguments | Required UTCP_ARG placeholder not provided | Validation error | +| Command Not Found | Command doesn't exist | Script execution error | +| Permission Denied | Insufficient permissions | Script execution error | +| Timeout | Script exceeded timeout | Async timeout error | +| Non-zero Exit | Script failed | Return stderr output | + +## Testing CLI Tools + +```python +import pytest +from utcp.utcp_client import UtcpClient + +@pytest.mark.asyncio +async def test_multi_command_cli_tool(): + client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "test_cli", + "call_template_type": "cli", + "commands": [ + { + "command": "echo UTCP_ARG_message_UTCP_END", + "append_to_final_output": false + }, + { + "command": "echo \"Previous: $CMD_0_OUTPUT\"" + } + ] + }] + }) + + result = await client.call_tool("test_cli.echo_chain", {"message": "hello"}) + assert "Previous: hello" in result +``` diff --git a/versioned_docs/version-1.0/protocols/http.md b/versioned_docs/version-1.0/protocols/http.md new file mode 100644 index 0000000..6340aeb --- /dev/null +++ b/versioned_docs/version-1.0/protocols/http.md @@ -0,0 +1,236 @@ +--- +id: http +title: HTTP Protocol +sidebar_position: 1 +--- + +# HTTP Protocol Plugin + +The HTTP protocol plugin enables UTCP to call tools via HTTP/HTTPS requests. This is the most commonly used protocol for REST APIs, webhooks, and web services. + +## Call Template Structure + +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "GET|POST|PUT|DELETE|PATCH", + "headers": { + "Content-Type": "application/json", + "User-Agent": "UTCP-Client/1.0" + }, + "query_params": { + "param1": "${variable1}", + "param2": "static_value" + }, + "body": { + "data": "${input_data}", + "timestamp": "${current_time}" + }, + "timeout": 30, + "verify_ssl": true, + "auth": { + "auth_type": "api_key|basic|oauth2", + "api_key": "${API_KEY}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +## Supported HTTP Methods + +| Method | Use Case | Body Support | +|--------|----------|--------------| +| `GET` | Retrieve data | No | +| `POST` | Create resources, submit data | Yes | +| `PUT` | Update/replace resources | Yes | +| `PATCH` | Partial updates | Yes | +| `DELETE` | Remove resources | Optional | + +## Authentication Methods + +### API Key Authentication +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +### Basic Authentication +```json +{ + "auth": { + "auth_type": "basic", + "username": "${USERNAME}", + "password": "${PASSWORD}" + } +} +``` + +### OAuth2 Authentication +```json +{ + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "read write" + } +} +``` + +## Variable Substitution + +Variables in call templates are substituted with values from: +- Tool call arguments: `${argument_name}` +- Environment variables: `${ENV_VAR}` +- Configuration variables: `${config.variable}` + +Example: +```json +{ + "url": "https://api.example.com/users/${user_id}", + "headers": { + "Authorization": "Bearer ${ACCESS_TOKEN}" + }, + "query_params": { + "format": "${output_format}", + "limit": "${max_results}" + } +} +``` + +## OpenAPI Integration + +The HTTP protocol plugin can automatically generate UTCP manuals from OpenAPI/Swagger specifications: + +1. **Automatic Discovery**: Point to an OpenAPI spec URL +2. **Schema Conversion**: Converts OpenAPI paths to UTCP tools +3. **Authentication Mapping**: Maps OpenAPI security schemes to UTCP auth +4. **Parameter Mapping**: Converts OpenAPI parameters to UTCP inputs + +Example OpenAPI to UTCP conversion: +```yaml +# OpenAPI Specification +paths: + /users/{id}: + get: + parameters: + - name: id + in: path + required: true + schema: + type: string +``` + +Becomes: +```json +{ + "name": "get_user", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/users/${id}", + "http_method": "GET" + }, + "inputs": { + "type": "object", + "properties": { + "id": {"type": "string"} + }, + "required": ["id"] + } +} +``` + +## Response Handling + +HTTP responses are processed based on: +- **Status Codes**: 2xx considered success, others as errors +- **Content-Type**: JSON, XML, text, and binary content support +- **Headers**: Response headers available in tool output +- **Error Mapping**: HTTP errors mapped to UTCP exceptions + +## Configuration Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `timeout` | number | 30 | Request timeout in seconds | +| `verify_ssl` | boolean | true | Verify SSL certificates | +| `follow_redirects` | boolean | true | Follow HTTP redirects | +| `max_redirects` | number | 5 | Maximum redirect hops | +| `retry_count` | number | 0 | Number of retry attempts | +| `retry_delay` | number | 1 | Delay between retries (seconds) | + +## Security Considerations + +### SSL/TLS Verification +- Always verify SSL certificates in production +- Use `verify_ssl: false` only for testing/development +- Consider certificate pinning for high-security applications + +### Authentication Security +- Store credentials in environment variables, not in configuration files +- Use OAuth2 for user-facing applications +- Rotate API keys regularly +- Implement proper token refresh for OAuth2 + +### Input Validation +- Validate all input parameters before substitution +- Sanitize user inputs to prevent injection attacks +- Use allowlists for acceptable parameter values +- Implement rate limiting on the tool provider side + +### Network Security +- Use HTTPS for all production communications +- Implement proper firewall rules +- Consider using VPNs or private networks for sensitive tools +- Monitor and log all HTTP requests for security analysis + +## Error Handling + +Common HTTP errors and their meanings: + +| Status Code | Error Type | Description | +|-------------|------------|-------------| +| 400 | Bad Request | Invalid request parameters | +| 401 | Unauthorized | Authentication required or failed | +| 403 | Forbidden | Access denied | +| 404 | Not Found | Resource doesn't exist | +| 429 | Rate Limited | Too many requests | +| 500 | Server Error | Internal server error | +| 503 | Service Unavailable | Service temporarily unavailable | + +## Best Practices + +### Performance +- Use connection pooling for multiple requests +- Implement appropriate timeouts +- Consider request/response compression +- Cache responses when appropriate + +### Reliability +- Implement retry logic with exponential backoff +- Handle network failures gracefully +- Use circuit breakers for unreliable services +- Monitor response times and error rates + +### Maintainability +- Use descriptive tool names and descriptions +- Document all required parameters +- Provide usage examples +- Version your APIs and update call templates accordingly + +## Language-Specific Implementation + +For implementation details and examples in your programming language: + +- **Multi-language**: [UTCP HTTP Protocol Examples](https://github.com/universal-tool-calling-protocol) - HTTP protocol examples across multiple languages +- **TypeScript**: [TypeScript HTTP Protocol Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/blob/main/docs/protocols/http.md) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) diff --git a/versioned_docs/version-1.0/protocols/index.md b/versioned_docs/version-1.0/protocols/index.md new file mode 100644 index 0000000..9a4f701 --- /dev/null +++ b/versioned_docs/version-1.0/protocols/index.md @@ -0,0 +1,93 @@ +--- +id: protocols +title: Protocols +sidebar_position: 4 +--- + +# Communication Protocol Plugins + +UTCP v1.0 features a modular, plugin-based architecture where different communication protocols are implemented as separate plugins. Each protocol plugin provides call templates and communication handlers for specific transport methods. + +## Available Protocol Plugins + +| Protocol | Plugin Package | Call Template | Use Cases | +|----------|----------------|---------------|-----------| +| **[HTTP](./http.md)** | `utcp-http` | `HttpCallTemplate` | REST APIs, webhooks, web services | +| **[WebSocket](./websocket.md)** | `utcp-websocket` | `WebSocketCallTemplate` | Real-time communication, streaming | +| **[CLI](./cli.md)** | `utcp-cli` | `CliCallTemplate` | Command-line tools, scripts | +| **[Server-Sent Events](./sse.md)** | `utcp-http` | `SseCallTemplate` | Event streaming, live updates | +| **[Text Files](./text.md)** | `utcp-text` | `TextCallTemplate` | File reading, static content | +| **[MCP](./mcp.md)** | `utcp-mcp` | `McpCallTemplate` | Model Context Protocol interop | + +## Plugin Architecture + +Each protocol plugin consists of: + +### Call Templates +Define how to structure calls for the specific protocol: +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "http_method": "POST", + "headers": {"Content-Type": "application/json"} +} +``` + +### Communication Protocols +Handle the actual communication logic for each protocol type. The implementation varies by programming language. + +## Installing Protocol Plugins + +Protocol plugins are available for different programming languages: + +```bash +# Example installation (Python) +pip install utcp-http utcp-cli utcp-websocket utcp-text utcp-mcp + +# Example installation (Node.js) +npm install @utcp/http @utcp/cli @utcp/websocket @utcp/text @utcp/mcp +``` + +For other languages, check the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) + +## Creating Custom Protocol Plugins + +You can extend UTCP with custom communication protocols by implementing the protocol interface in your chosen language. Each implementation must: + +1. **Define Call Templates**: Specify the structure for protocol-specific calls +2. **Implement Communication Handler**: Handle the actual protocol communication +3. **Register the Protocol**: Make it available to the UTCP client + +Example call template structure: +```json +{ + "call_template_type": "custom", + "custom_field": "value", + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}" + } +} +``` + +## Protocol Selection Guide + +Choose the right protocol plugin based on your needs: + +- **HTTP**: Most common for REST APIs and web services +- **WebSocket**: Real-time bidirectional communication +- **CLI**: Wrapping existing command-line tools +- **SSE**: Server-sent events for streaming data +- **Text**: Reading configuration files or static content +- **MCP**: Interoperability with Model Context Protocol tools + +## Language-Specific Documentation + +For implementation details and examples in your programming language: + +- **Multi-language**: [UTCP Implementation Examples](https://github.com/universal-tool-calling-protocol) - Examples across Python, TypeScript, Go, and other languages +- **TypeScript**: [TypeScript UTCP Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/tree/main/docs) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) + +For detailed information about each protocol plugin, see the individual protocol documentation pages. diff --git a/versioned_docs/version-1.0/protocols/mcp.md b/versioned_docs/version-1.0/protocols/mcp.md new file mode 100644 index 0000000..13da0f6 --- /dev/null +++ b/versioned_docs/version-1.0/protocols/mcp.md @@ -0,0 +1,266 @@ +--- +id: mcp +title: MCP Protocol +sidebar_position: 6 +--- + +# MCP Protocol Plugin + +The MCP (Model Context Protocol) plugin provides interoperability between UTCP and existing MCP servers, enabling gradual migration from MCP to UTCP while maintaining compatibility with existing MCP tools. + +## Call Template Structure + +```json +{ + "call_template_type": "mcp", + "server_config": { + "command": "node", + "args": ["mcp-server.js"], + "working_directory": "/app/mcp", + "env": { + "NODE_ENV": "production", + "LOG_LEVEL": "info" + }, + "timeout": 30 + }, + "connection_timeout": 10, + "request_timeout": 30 +} +``` + +## Server Configuration + +### Command-based Servers +```json +{ + "server_config": { + "command": "python", + "args": ["-m", "mcp_server", "--config", "config.json"], + "working_directory": "/app", + "env": { + "PYTHONPATH": "/app/lib", + "API_KEY": "${MCP_API_KEY}" + } + } +} +``` + +### HTTP-based Servers +```json +{ + "server_config": { + "transport": "http", + "url": "http://localhost:8080/mcp", + "headers": { + "Authorization": "Bearer ${MCP_TOKEN}" + } + } +} +``` + +## Migration Strategy + +The MCP protocol plugin enables a gradual migration path from MCP to native UTCP protocols: + +### Phase 1: MCP Integration +- Use existing MCP servers through UTCP +- No changes to MCP server code required +- UTCP client can call MCP tools seamlessly + +### Phase 2: Hybrid Approach +- Some tools use native UTCP protocols +- Legacy tools continue using MCP +- Gradual migration of high-value tools + +### Phase 3: Full Migration +- All tools use native UTCP protocols +- MCP servers deprecated +- Simplified architecture + +## Tool Discovery + +MCP servers expose tools through the standard MCP protocol. The UTCP MCP plugin: + +1. **Connects** to the MCP server using stdio or HTTP transport +2. **Discovers** available tools via MCP's `tools/list` method +3. **Maps** MCP tool definitions to UTCP tool format +4. **Registers** tools in the UTCP client + +## Request/Response Mapping + +### MCP to UTCP Tool Mapping +```json +// MCP Tool Definition +{ + "name": "read_file", + "description": "Read contents of a file", + "inputSchema": { + "type": "object", + "properties": { + "path": {"type": "string"} + }, + "required": ["path"] + } +} + +// UTCP Tool (after mapping) +{ + "name": "read_file", + "description": "Read contents of a file", + "inputs": { + "type": "object", + "properties": { + "path": {"type": "string"} + }, + "required": ["path"] + }, + "tool_call_template": { + "call_template_type": "mcp", + "server_config": {...} + } +} +``` + +### Request Flow +1. UTCP client receives tool call +2. MCP plugin formats request as MCP `tools/call` +3. Request sent to MCP server +4. MCP server processes and responds +5. Response mapped back to UTCP format + +## Authentication and Security + +### Server Authentication +```json +{ + "server_config": { + "command": "secure-mcp-server", + "env": { + "MCP_AUTH_TOKEN": "${MCP_SERVER_TOKEN}", + "MCP_CLIENT_ID": "${MCP_CLIENT_ID}" + } + } +} +``` + +### Transport Security +- **stdio**: Inherits process security model +- **HTTP**: Use HTTPS and proper authentication headers +- **WebSocket**: Use WSS and authentication tokens + +## Error Handling + +### Connection Errors +- Server startup failures +- Network connectivity issues +- Authentication failures +- Timeout errors + +### Protocol Errors +- Invalid MCP messages +- Unsupported MCP features +- Tool execution failures +- Resource access errors + +### Error Mapping +MCP errors are mapped to UTCP exceptions: +- `InvalidRequest` → `ValidationError` +- `MethodNotFound` → `ToolNotFoundError` +- `InternalError` → `ToolCallError` + +## Performance Considerations + +### Connection Management +- Persistent connections for stdio transport +- Connection pooling for HTTP transport +- Automatic reconnection on failures +- Graceful shutdown handling + +### Request Optimization +- Batch multiple tool calls when possible +- Cache tool discovery results +- Implement request timeouts +- Monitor response times + +## Limitations + +### MCP Feature Support +Not all MCP features are supported through UTCP: +- **Resources**: Not directly mapped to UTCP tools +- **Prompts**: Not supported in UTCP model +- **Sampling**: Not applicable to tool calling + +### Protocol Differences +- MCP's bidirectional communication vs UTCP's request/response +- MCP's resource model vs UTCP's tool-only model +- Different authentication mechanisms + +## Configuration Examples + +### Development Setup +```json +{ + "manual_call_templates": [{ + "name": "dev_mcp", + "call_template_type": "mcp", + "server_config": { + "command": "node", + "args": ["dev-server.js"], + "env": {"NODE_ENV": "development"} + }, + "connection_timeout": 5, + "request_timeout": 10 + }] +} +``` + +### Production Setup +```json +{ + "manual_call_templates": [{ + "name": "prod_mcp", + "call_template_type": "mcp", + "server_config": { + "transport": "http", + "url": "https://mcp.example.com/api", + "headers": { + "Authorization": "Bearer ${MCP_PROD_TOKEN}", + "X-Client-Version": "1.0.0" + } + }, + "connection_timeout": 30, + "request_timeout": 60 + }] +} +``` + +## Best Practices + +### Migration Planning +1. **Inventory** existing MCP servers and tools +2. **Prioritize** tools for migration based on usage +3. **Test** MCP integration thoroughly +4. **Monitor** performance and reliability +5. **Migrate** incrementally to native UTCP protocols + +### Monitoring and Debugging +- Enable debug logging for MCP communication +- Monitor server health and response times +- Track tool usage patterns +- Log authentication failures +- Set up alerts for connection issues + +### Security +- Use secure transport methods (HTTPS, WSS) +- Implement proper authentication +- Validate all inputs and outputs +- Monitor for suspicious activity +- Keep MCP servers updated + +## Language-Specific Implementation + +For implementation details and examples in your programming language: + +- **Multi-language**: [UTCP MCP Protocol Examples](https://github.com/universal-tool-calling-protocol) - MCP protocol examples across multiple languages +- **TypeScript**: [TypeScript MCP Protocol Documentation](https://github.com/universal-tool-calling-protocol/typescript-utcp/blob/main/docs/protocols/mcp.md) +- **Other languages**: Check respective repositories in the [UTCP GitHub organization](https://github.com/universal-tool-calling-protocol) diff --git a/versioned_docs/version-1.0/protocols/sse.md b/versioned_docs/version-1.0/protocols/sse.md new file mode 100644 index 0000000..93618e9 --- /dev/null +++ b/versioned_docs/version-1.0/protocols/sse.md @@ -0,0 +1,386 @@ +--- +id: sse +title: Server-Sent Events (SSE) +sidebar_position: 3 +--- + +# Server-Sent Events (SSE) + +The Server-Sent Events protocol plugin (`utcp-http`) enables UTCP to receive real-time streaming data from HTTP servers. SSE is perfect for tools that need to stream live updates, notifications, or continuous data feeds. + +## Installation + +SSE support is included with the HTTP plugin: + +```bash +# Example installation (Python) +pip install utcp-http + +# Example installation (Node.js) +npm install @utcp/http +``` + +## Call Template Structure + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "headers": { + "Authorization": "Bearer ${API_TOKEN}", + "Accept": "text/event-stream" + }, + "timeout": 60, + "max_events": 10, + "event_filter": { + "type": "data_update" + } +} +``` + +## Configuration Options + +The Server-Sent Events (SSE) call template enables real-time streaming data from HTTP endpoints. For complete field specifications and validation rules, see the [SSE Call Template API Reference](../api/plugins/communication_protocols/http/src/utcp_http/sse_call_template.md). +| `reconnect` | boolean | Auto-reconnect on connection loss (default: true) | +| `reconnect_delay` | number | Delay between reconnection attempts (default: 3) | + +## Authentication + +SSE uses standard HTTP authentication methods: + +### Bearer Token + +```json +{ + "headers": { + "Authorization": "Bearer ${ACCESS_TOKEN}" + } +} +``` + +### API Key + +```json +{ + "headers": { + "X-API-Key": "${API_KEY}" + } +} +``` + +### Query Parameter Auth + +```json +{ + "query_params": { + "token": "${API_TOKEN}", + "user_id": "${USER_ID}" + } +} +``` + +## Event Handling + +### Basic Event Stream + +```json +{ + "name": "stream_notifications", + "description": "Stream real-time notifications", + "inputs": { + "type": "object", + "properties": { + "user_id": {"type": "string"} + }, + "required": ["user_id"] + }, + "tool_call_template": { + "call_template_type": "sse", + "url": "https://api.example.com/notifications/stream", + "query_params": { + "user_id": "${user_id}" + }, + "headers": { + "Authorization": "Bearer ${ACCESS_TOKEN}" + }, + "timeout": 300 + } +} +``` + +### Filtered Events + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "event_filter": { + "type": "order_update", + "status": ["completed", "cancelled"] + }, + "max_events": 5 +} +``` + +## Examples + +### Stock Price Stream + +```json +{ + "name": "stream_stock_prices", + "description": "Stream real-time stock price updates", + "inputs": { + "type": "object", + "properties": { + "symbols": { + "type": "array", + "items": {"type": "string"} + }, + "duration": {"type": "number", "default": 60} + }, + "required": ["symbols"] + }, + "tool_call_template": { + "call_template_type": "sse", + "url": "https://api.stocks.com/stream", + "query_params": { + "symbols": "${symbols}", + "format": "json" + }, + "headers": { + "Authorization": "Bearer ${STOCK_API_KEY}" + }, + "timeout": "${duration}", + "event_filter": { + "type": "price_update" + } + } +} +``` + +### Log Monitoring + +```json +{ + "name": "monitor_logs", + "description": "Monitor application logs in real-time", + "inputs": { + "type": "object", + "properties": { + "service": {"type": "string"}, + "level": {"type": "string", "enum": ["error", "warn", "info", "debug"]} + }, + "required": ["service"] + }, + "tool_call_template": { + "call_template_type": "sse", + "url": "https://logs.example.com/stream", + "query_params": { + "service": "${service}", + "level": "${level}" + }, + "headers": { + "X-API-Key": "${LOG_API_KEY}" + }, + "timeout": 600, + "max_events": 100 + } +} +``` + +### System Metrics Stream + +```json +{ + "name": "stream_metrics", + "description": "Stream system performance metrics", + "inputs": { + "type": "object", + "properties": { + "metrics": { + "type": "array", + "items": {"type": "string"} + }, + "interval": {"type": "number", "default": 5} + }, + "required": ["metrics"] + }, + "tool_call_template": { + "call_template_type": "sse", + "url": "https://monitoring.example.com/metrics/stream", + "query_params": { + "metrics": "${metrics}", + "interval": "${interval}" + }, + "headers": { + "Authorization": "Bearer ${MONITORING_TOKEN}" + }, + "timeout": 300, + "reconnect": true, + "reconnect_delay": 5 + } +} +``` + +## Event Format + +SSE events follow the standard format: + +``` +event: message +data: {"type": "update", "value": 123} +id: event-123 +retry: 3000 + +event: heartbeat +data: {"timestamp": "2024-01-15T10:30:00Z"} + +data: {"message": "Simple data without event type"} +``` + +### Parsed Event Structure + +```json +{ + "event": "message", + "data": {"type": "update", "value": 123}, + "id": "event-123", + "retry": 3000, + "timestamp": "2024-01-15T10:30:00Z" +} +``` + +## Response Handling + +### Single Event Response + +```json +{ + "events": [ + { + "event": "data", + "data": {"result": "success"}, + "id": "1", + "timestamp": "2024-01-15T10:30:00Z" + } + ], + "total_events": 1, + "duration": 2.5 +} +``` + +### Multiple Events Response + +```json +{ + "events": [ + { + "event": "start", + "data": {"status": "processing"}, + "id": "1" + }, + { + "event": "progress", + "data": {"percent": 50}, + "id": "2" + }, + { + "event": "complete", + "data": {"result": "success"}, + "id": "3" + } + ], + "total_events": 3, + "duration": 15.2 +} +``` + +## Error Handling + +| Error Type | Description | Handling | +|------------|-------------|----------| +| Connection Failed | Cannot connect to SSE endpoint | Raise `SSEConnectionError` | +| Stream Timeout | No events received within timeout | Return partial results | +| Parse Error | Invalid SSE event format | Skip malformed events | +| Authentication Failed | Invalid credentials | Raise `SSEAuthError` | +| Server Error | HTTP 5xx response | Raise `SSEServerError` | + +## Best Practices + +1. **Set Appropriate Timeouts**: Configure timeouts based on expected data frequency +2. **Handle Reconnections**: Enable auto-reconnect for long-running streams +3. **Filter Events**: Use event filters to reduce unnecessary data processing +4. **Monitor Performance**: Track event rates and processing times +5. **Validate Data**: Validate incoming event data against expected schemas +6. **Handle Backpressure**: Implement buffering for high-frequency events +7. **Graceful Shutdown**: Properly close streams when done + +## Advanced Features + +### Custom Event Parsing + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "event_parser": { + "format": "json", + "extract_fields": ["timestamp", "level", "message"] + } +} +``` + +### Event Aggregation + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/metrics", + "aggregation": { + "window": 10, + "function": "average", + "field": "value" + } +} +``` + +### Conditional Termination + +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "termination_condition": { + "event_type": "complete", + "data_field": "status", + "value": "finished" + } +} +``` + +## Common Use Cases + +- **Real-time Dashboards**: Live metrics, status updates +- **Notifications**: User alerts, system notifications +- **Log Streaming**: Application logs, audit trails +- **Progress Tracking**: Long-running task progress +- **Live Data Feeds**: News, social media, sensor data +- **Chat Applications**: Message streams, typing indicators + +## Protocol Comparison + +| Feature | SSE | WebSocket | HTTP Polling | +|---------|-----|-----------|--------------| +| Server-to-Client | ✅ | ✅ | ✅ | +| Client-to-Server | ❌ | ✅ | ✅ | +| Auto-Reconnect | ✅ | Manual | Manual | +| Overhead | Low | Low | High | +| Browser Support | ✅ | ✅ | ✅ | +| Simplicity | High | Medium | High | + +## Related Protocols + +- [HTTP](./http.md) - For request/response patterns +- [WebSocket](./websocket.md) - For bidirectional communication +- [Streamable HTTP](./streamable-http.md) - For chunked HTTP responses diff --git a/versioned_docs/version-1.0/protocols/streamable-http.md b/versioned_docs/version-1.0/protocols/streamable-http.md new file mode 100644 index 0000000..3ed9e6e --- /dev/null +++ b/versioned_docs/version-1.0/protocols/streamable-http.md @@ -0,0 +1,105 @@ +--- +id: streamable-http +title: Streamable HTTP Protocol +sidebar_position: 2 +--- + +# Streamable HTTP Protocol + +The Streamable HTTP protocol plugin (`utcp-http`) enables UTCP to handle large HTTP responses by streaming them in chunks. This is ideal for tools that return large datasets, files, or progressive results that don't fit into a single response payload. It leverages HTTP Chunked Transfer Encoding. + +## Call Template Structure + +```json +{ + "call_template_type": "streamable_http", + "url": "https://api.example.com/download/large-file", + "http_method": "GET", + "headers": { + "Accept": "application/octet-stream" + }, + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "Authorization", + "location": "header" + }, + "chunk_size": 8192, + "timeout": 300000, + "body_field": "request_data", + "header_fields": ["custom_header_arg"] +} +``` + +## Configuration Options + +The Streamable HTTP call template provides a way to configure streaming from HTTP endpoints. + +| Option | Type | Default | Description | +|---|---|---|---| +| `url` | string | **Required** | The streaming HTTP endpoint URL. Supports path parameters like `/users/{user_id}`. | +| `http_method` | string | `GET` | The HTTP method to use. Supported methods are `GET` and `POST`. | +| `content_type` | string | `application/octet-stream` | The `Content-Type` header to set for the request, especially when `body_field` is used. | +| `chunk_size` | integer | `4096` | The size of each data chunk in bytes to read from the stream. | +| `timeout` | integer | `60000` | Request timeout in milliseconds. | +| `headers` | object | `null` | Optional static headers to include in every request. | +| `auth` | object | `null` | Optional authentication configuration. See [HTTP Authentication](./http.md#authentication-methods). | +| `body_field` | string | `null` | The name of a single tool argument to be sent as the HTTP request body. | +| `header_fields` | array | `null` | A list of tool argument names to be sent as request headers. | + +## Response Handling + +The protocol processes the incoming stream based on the `Content-Type` header of the response: + +- **`application/x-ndjson`**: The stream is parsed as Newline Delimited JSON. Each line is yielded as a separate JSON object. +- **`application/octet-stream`**: The stream is yielded in binary chunks of the specified `chunk_size`. +- **`application/json`**: The entire response is buffered and yielded as a single JSON object. This is for endpoints that stream a single, large JSON document. +- **Other Types**: For any other `Content-Type`, the response is treated as a binary stream and yielded in chunks of `chunk_size`. + +## Authentication + +Streamable HTTP supports the same authentication methods as the standard HTTP protocol, including API Key, Basic Auth, and OAuth2. The configuration is identical. + +For more details, see the [HTTP Authentication Methods](./http.md#authentication-methods) documentation. + +## Variable Substitution + +Path parameters, query parameters, headers, and authentication fields all support variable substitution from tool arguments and environment variables, following the same syntax as the standard HTTP protocol. + +Example: +```json +{ + "url": "https://api.example.com/files/{file_id}/download", + "headers": { + "Authorization": "Bearer ${ACCESS_TOKEN}" + } +} +``` +Here, `{file_id}` is substituted from a tool argument, and `${ACCESS_TOKEN}` is substituted from an environment or configuration variable. + +For more details, see the [HTTP Variable Substitution](./http.md#variable-substitution) documentation. + +## Security Considerations + +- **SSL/TLS**: It is strongly recommended to use `https://` endpoints to protect data in transit. The implementation enforces HTTPS or localhost connections by default. +- **Authentication**: Never hardcode credentials. Use variable substitution to inject secrets from a secure source (e.g., environment variables). +- **Input Sanitization**: Ensure that any arguments used in URL path parameters or query strings are properly validated and sanitized to prevent injection attacks. + +## Error Handling + +Errors are handled similarly to the standard HTTP protocol: + +| Status Code | Error Type | Description | +|---|---|---| +| 400 | Bad Request | Invalid request parameters or body. | +| 401 | Unauthorized | Authentication failed or is required. | +| 403 | Forbidden | The authenticated user does not have permission. | +| 404 | Not Found | The requested resource or endpoint does not exist. | +| 5xx | Server Error | An error occurred on the server side. | + +Connection errors, timeouts, and other network issues will also be raised as exceptions. + +## Related Protocols + +- [HTTP](./http.md) - For standard request/response interactions. +- [Server-Sent Events (SSE)](./sse.md) - For unidirectional, real-time event streams from server to client. \ No newline at end of file diff --git a/versioned_docs/version-1.0/protocols/text.md b/versioned_docs/version-1.0/protocols/text.md new file mode 100644 index 0000000..dcca388 --- /dev/null +++ b/versioned_docs/version-1.0/protocols/text.md @@ -0,0 +1,437 @@ +--- +id: text +title: Text Protocol +sidebar_position: 5 +--- + +# Text Protocol + +The Text protocol plugin (`utcp-text`) enables UTCP to read and process text files from local filesystem or remote URLs. This is useful for tools that need to access documentation, configuration files, logs, or any text-based data. + +## Installation + +```bash +# Example installation (Python) +pip install utcp-text + +# Example installation (Node.js) +npm install @utcp/text +``` + +## Call Template Structure + +```json +{ + "call_template_type": "text", + "file_path": "/path/to/file.txt", + "encoding": "utf-8", + "max_size": 1048576, + "line_range": { + "start": 1, + "end": 100 + } +} +``` + +## Configuration Options + +The Text call template enables reading and processing text files from local filesystem or URLs. For complete field specifications and validation rules, see the [Text Call Template API Reference](../api/plugins/communication_protocols/text/src/utcp_text/text_call_template.md). + +## File Sources + +### Local Files + +```json +{ + "call_template_type": "text", + "file_path": "/var/log/application.log", + "encoding": "utf-8" +} +``` + +### Remote URLs + +```json +{ + "call_template_type": "text", + "file_path": "https://example.com/config.txt", + "max_size": 512000 +} +``` + +### Variable Substitution + +```json +{ + "call_template_type": "text", + "file_path": "/data/${filename}", + "encoding": "${file_encoding}" +} +``` + +## Examples + +### Read Configuration File + +```json +{ + "name": "read_config", + "description": "Read application configuration file", + "inputs": { + "type": "object", + "properties": { + "config_name": {"type": "string"} + }, + "required": ["config_name"] + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "/etc/app/${config_name}.conf", + "encoding": "utf-8", + "max_size": 65536 + } +} +``` + +### Read Log File with Line Range + +```json +{ + "name": "read_recent_logs", + "description": "Read recent log entries", + "inputs": { + "type": "object", + "properties": { + "log_file": {"type": "string"}, + "lines": {"type": "number", "default": 100} + }, + "required": ["log_file"] + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "/var/log/${log_file}", + "line_range": { + "start": -${lines}, + "end": -1 + } + } +} +``` + +### Read Remote Documentation + +```json +{ + "name": "fetch_documentation", + "description": "Fetch documentation from remote URL", + "inputs": { + "type": "object", + "properties": { + "doc_url": {"type": "string"}, + "section": {"type": "string"} + }, + "required": ["doc_url"] + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "${doc_url}", + "pattern": "(?s)## ${section}.*?(?=## |$)", + "max_size": 2097152 + } +} +``` + +### Search in File + +```json +{ + "name": "search_in_file", + "description": "Search for pattern in text file", + "inputs": { + "type": "object", + "properties": { + "file_path": {"type": "string"}, + "search_pattern": {"type": "string"} + }, + "required": ["file_path", "search_pattern"] + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "${file_path}", + "pattern": "${search_pattern}", + "transform": "strip" + } +} +``` + +## Line Range Options + +### Absolute Line Numbers + +```json +{ + "line_range": { + "start": 10, + "end": 50 + } +} +``` + +### Relative to End (Tail) + +```json +{ + "line_range": { + "start": -100, + "end": -1 + } +} +``` + +### From Start (Head) + +```json +{ + "line_range": { + "start": 1, + "end": 100 + } +} +``` + +## Pattern Matching + +### Simple Text Search + +```json +{ + "pattern": "ERROR" +} +``` + +### Regex Pattern + +```json +{ + "pattern": "\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2} ERROR.*" +} +``` + +### Multi-line Pattern + +```json +{ + "pattern": "(?s)START.*?END" +} +``` + +## Content Transformations + +### Case Transformations + +```json +{ + "transform": "upper" // Convert to uppercase +} +``` + +```json +{ + "transform": "lower" // Convert to lowercase +} +``` + +### Whitespace Handling + +```json +{ + "transform": "strip" // Remove leading/trailing whitespace +} +``` + +### Custom Transformations + +```json +{ + "transform": "normalize_whitespace" // Normalize all whitespace +} +``` + +## Response Format + +### Successful Read + +```json +{ + "content": "File content here...", + "metadata": { + "file_path": "/path/to/file.txt", + "size": 1024, + "lines": 25, + "encoding": "utf-8", + "last_modified": "2024-01-15T10:30:00Z" + } +} +``` + +### Filtered Content + +```json +{ + "content": "Matching lines...", + "metadata": { + "file_path": "/path/to/file.txt", + "total_lines": 1000, + "matched_lines": 5, + "pattern": "ERROR", + "line_range": {"start": 1, "end": 100} + } +} +``` + +## Error Handling + +| Error Type | Description | Handling | +|------------|-------------|----------| +| File Not Found | File doesn't exist | Raise `FileNotFoundError` | +| Permission Denied | No read permission | Raise `PermissionError` | +| File Too Large | Exceeds max_size limit | Raise `FileSizeError` | +| Encoding Error | Invalid file encoding | Raise `EncodingError` | +| Network Error | URL fetch failed | Raise `NetworkError` | + +## Security Considerations + +### Path Traversal Prevention + +```json +{ + "call_template_type": "text", + "file_path": "/safe/directory/${filename}", + "allowed_paths": ["/safe/directory/"] +} +``` + +### File Size Limits + +```json +{ + "max_size": 1048576 // 1MB limit +} +``` + +### URL Restrictions + +```json +{ + "allowed_domains": ["example.com", "docs.company.com"] +} +``` + +## Best Practices + +1. **Set Size Limits**: Always set appropriate max_size limits +2. **Validate Paths**: Validate file paths to prevent directory traversal +3. **Handle Encoding**: Specify encoding explicitly for non-UTF-8 files +4. **Use Line Ranges**: Use line ranges for large files to improve performance +5. **Pattern Efficiency**: Use efficient regex patterns for content filtering +6. **Cache Results**: Cache frequently accessed files +7. **Monitor Access**: Log file access for security auditing + +## Advanced Features + +### Conditional Reading + +```json +{ + "call_template_type": "text", + "file_path": "/var/log/app.log", + "condition": { + "modified_since": "2024-01-15T00:00:00Z" + } +} +``` + +### Multi-file Reading + +```json +{ + "call_template_type": "text", + "file_paths": [ + "/etc/app/config1.txt", + "/etc/app/config2.txt" + ], + "merge_strategy": "concatenate" +} +``` + +### Streaming Large Files + +```json +{ + "call_template_type": "text", + "file_path": "/var/log/huge.log", + "streaming": true, + "chunk_size": 8192 +} +``` + +## Common Use Cases + +- **Configuration Management**: Reading config files, environment files +- **Log Analysis**: Processing application logs, system logs +- **Documentation**: Accessing README files, API docs, manuals +- **Data Processing**: Reading CSV, JSON, XML text files +- **Template Processing**: Reading template files for generation +- **Code Analysis**: Reading source code files for analysis +- **Monitoring**: Reading status files, health check files + +## Performance Considerations + +| File Size | Recommended Approach | +|-----------|---------------------| +| < 1MB | Read entire file | +| 1MB - 10MB | Use line ranges | +| 10MB - 100MB | Use streaming | +| > 100MB | Use external tools | + +## Integration Examples + +### With HTTP Protocol + +```json +{ + "name": "process_uploaded_file", + "description": "Process uploaded text file", + "inputs": { + "type": "object", + "properties": { + "file_url": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "text", + "file_path": "${file_url}", + "max_size": 5242880 + } +} +``` + +### With CLI Protocol + +```json +{ + "name": "analyze_log_file", + "description": "Analyze log file with external tool", + "inputs": { + "type": "object", + "properties": { + "log_path": {"type": "string"} + } + }, + "tool_call_template": { + "call_template_type": "cli", + "command": "log-analyzer", + "args": ["--file", "${log_path}", "--format", "json"] + } +} +``` diff --git a/versioned_docs/version-1.0/protocols/websocket.md b/versioned_docs/version-1.0/protocols/websocket.md new file mode 100644 index 0000000..5ce7789 --- /dev/null +++ b/versioned_docs/version-1.0/protocols/websocket.md @@ -0,0 +1,355 @@ +--- +id: websocket +title: WebSocket Protocol +sidebar_position: 2 +--- + +# WebSocket Protocol + +The WebSocket protocol plugin (`utcp-websocket`) enables UTCP to communicate with WebSocket servers for real-time, bidirectional communication. This is ideal for tools that require persistent connections or real-time updates. + +## Installation + +```bash +# Example installation (Python) +pip install utcp-websocket + +# Example installation (Node.js) +npm install @utcp/websocket +``` + +## Call Template Structure + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": { + "type": "request", + "action": "${action}", + "data": "${data}" + }, + "connection_timeout": 10, + "response_timeout": 30, + "auth": { + "auth_type": "api_key", + "api_key": "${WS_API_KEY}", + "location": "query" + } +} +``` + +## Configuration Options + +The WebSocket call template enables real-time communication with WebSocket servers. For complete field specifications and validation rules, see the WebSocket Call Template API Reference (WIP). +| `expected_responses` | number | Number of expected response messages (default: 1) | +| `ping_interval` | number | Ping interval in seconds (default: 30) | + +## Authentication + +WebSocket authentication can be handled in several ways: + +### Query Parameter Authentication + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "token", + "location": "query" + } +} +``` + +### Header Authentication + +```json +{ + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "Authorization", + "location": "header" + } +} +``` + +### Message-based Authentication + +```json +{ + "message": { + "type": "auth", + "token": "${API_KEY}" + } +} +``` + +## Message Formats + +### JSON Messages + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": { + "id": "{{uuid}}", + "method": "getData", + "params": { + "query": "${query}", + "limit": 10 + } + } +} +``` + +### Text Messages + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": "GET_DATA:${query}" +} +``` + +### Binary Messages + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": { + "type": "binary", + "data": "${base64_data}" + } +} +``` + +## Examples + +### Real-time Data Subscription + +```json +{ + "name": "subscribe_stock_price", + "description": "Subscribe to real-time stock price updates", + "inputs": { + "type": "object", + "properties": { + "symbol": {"type": "string"}, + "duration": {"type": "number", "default": 60} + }, + "required": ["symbol"] + }, + "tool_call_template": { + "call_template_type": "websocket", + "url": "wss://api.stocks.com/ws", + "message": { + "action": "subscribe", + "symbol": "${symbol}", + "type": "price" + }, + "response_timeout": "${duration}", + "expected_responses": -1, + "close_after_response": false + } +} +``` + +### Chat Bot Integration + +```json +{ + "name": "send_chat_message", + "description": "Send a message to the chat bot", + "inputs": { + "type": "object", + "properties": { + "message": {"type": "string"}, + "user_id": {"type": "string"} + }, + "required": ["message", "user_id"] + }, + "tool_call_template": { + "call_template_type": "websocket", + "url": "wss://chat.example.com/ws", + "message": { + "type": "message", + "user_id": "${user_id}", + "content": "${message}", + "timestamp": "{{now}}" + }, + "headers": { + "Authorization": "Bearer ${CHAT_TOKEN}" + } + } +} +``` + +### IoT Device Control + +```json +{ + "name": "control_device", + "description": "Send control commands to IoT device", + "inputs": { + "type": "object", + "properties": { + "device_id": {"type": "string"}, + "command": {"type": "string"}, + "value": {"type": "number"} + }, + "required": ["device_id", "command"] + }, + "tool_call_template": { + "call_template_type": "websocket", + "url": "wss://iot.example.com/device/${device_id}", + "message": { + "command": "${command}", + "value": "${value}", + "timestamp": "{{now}}" + }, + "connection_timeout": 5, + "response_timeout": 10 + } +} +``` + +## Connection Management + +### Persistent Connections + +For tools that need to maintain persistent connections: + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": {"action": "ping"}, + "close_after_response": false, + "ping_interval": 30 +} +``` + +### Connection Pooling + +The WebSocket protocol automatically manages connection pooling for efficiency: + +- Reuses connections to the same endpoint +- Handles connection lifecycle automatically +- Implements reconnection logic for dropped connections + +## Response Handling + +### Single Response + +```json +{ + "expected_responses": 1, + "close_after_response": true +} +``` + +### Multiple Responses + +```json +{ + "expected_responses": 5, + "response_timeout": 60 +} +``` + +### Streaming Responses + +```json +{ + "expected_responses": -1, + "response_timeout": 300, + "close_after_response": false +} +``` + +## Error Handling + +| Error Type | Description | Handling | +|------------|-------------|----------| +| Connection Failed | Cannot establish WebSocket connection | Raise `WebSocketConnectionError` | +| Authentication Failed | WebSocket handshake authentication failed | Raise `WebSocketAuthError` | +| Timeout | No response within timeout period | Raise `WebSocketTimeoutError` | +| Protocol Error | Invalid WebSocket protocol usage | Raise `WebSocketProtocolError` | +| Connection Closed | Server closed connection unexpectedly | Raise `WebSocketClosedError` | + +## Best Practices + +1. **Use Secure WebSockets**: Always use `wss://` for production +2. **Handle Reconnections**: Implement retry logic for connection failures +3. **Set Appropriate Timeouts**: Configure timeouts based on expected response times +4. **Validate Messages**: Validate both outgoing and incoming messages +5. **Monitor Connections**: Track connection health and performance +6. **Implement Heartbeats**: Use ping/pong for connection health checks +7. **Handle Backpressure**: Manage message queuing for high-throughput scenarios + +## Advanced Features + +### Message Filtering + +Filter incoming messages based on criteria: + +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "message": {"subscribe": "all"}, + "message_filter": { + "type": "stock_price", + "symbol": "${symbol}" + } +} +``` + +### Custom Headers + +Include custom headers in the WebSocket handshake: + +```json +{ + "headers": { + "User-Agent": "UTCP-Client/1.0", + "X-Client-ID": "${CLIENT_ID}", + "Authorization": "Bearer ${TOKEN}" + } +} +``` + +### Compression + +Enable WebSocket compression: + +```json +{ + "compression": "deflate", + "compression_threshold": 1024 +} +``` + +## Common Use Cases + +- **Real-time Data**: Stock prices, sensor data, live metrics +- **Chat Applications**: Messaging, notifications, presence +- **Gaming**: Real-time game state, multiplayer coordination +- **IoT Control**: Device commands, status updates +- **Live Updates**: News feeds, social media streams +- **Collaborative Tools**: Document editing, shared whiteboards + +## Protocol Comparison + +| Feature | WebSocket | HTTP | SSE | +|---------|-----------|------|-----| +| Bidirectional | ✅ | ❌ | ❌ | +| Real-time | ✅ | ❌ | ✅ | +| Persistent | ✅ | ❌ | ✅ | +| Overhead | Low | High | Medium | +| Complexity | Medium | Low | Low | diff --git a/versioned_docs/version-1.0/security.md b/versioned_docs/version-1.0/security.md index b182938..fbf10f6 100644 --- a/versioned_docs/version-1.0/security.md +++ b/versioned_docs/version-1.0/security.md @@ -6,123 +6,439 @@ sidebar_position: 6 # Security Considerations -Security is a critical aspect of any protocol that enables tool access and execution. This section outlines key security considerations when implementing and using UTCP. +Security is critical when enabling direct tool access through UTCP. This guide covers security considerations specific to UTCP's "manual" approach and provides practical guidance for secure implementations. -## Authentication +## UTCP Security Model -UTCP supports several authentication methods across different communication protocol types: +### Direct Communication Implications -### API Key Authentication +UTCP's direct communication model has unique security characteristics: + +**Advantages:** +- No middleman to compromise +- Native security controls remain intact +- Reduced attack surface (no proxy servers) +- Existing monitoring and logging continue to work + +**Considerations:** +- Clients must handle multiple authentication methods +- Manual endpoints become discovery targets +- Variable substitution introduces injection risks + +### Manual Discovery Security + +Secure your UTCP manual endpoints: + +**Best Practices:** +- Implement authentication for manual discovery endpoints +- Validate discovery access tokens +- Return only tools the client is authorized to see +- Filter tools based on user permissions +- Log all manual discovery attempts + +## Authentication & Authorization + +### Enhanced Authentication Examples + +#### API Key with Rotation ```json { "auth": { "auth_type": "api_key", - "api_key": "YOUR_API_KEY", + "api_key": "${API_KEY}", "var_name": "X-API-Key", "location": "header" } } ``` -The `location` field specifies where the API key is placed, and can be `header`, `query`, or `cookie`. +**Secure implementation:** +- Use rotating API keys with current and next key support +- Implement automatic key rotation based on time +- Store keys securely in environment variables +- Validate keys before processing requests -### Basic Authentication +#### OAuth2 with Scope Validation ```json { "auth": { - "auth_type": "basic", - "username": "user", - "password": "pass" + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token", + "scope": "tools:read tools:execute" } } ``` -### OAuth2 Authentication +#### Per-Tool Authorization ```json { - "auth": { - "auth_type": "oauth2", - "client_id": "YOUR_CLIENT_ID", - "client_secret": "YOUR_CLIENT_SECRET", - "token_url": "https://auth.example.com/token", - "scope": "read:tools" + "name": "admin_tool", + "description": "Administrative operations", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/admin/action", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "${ADMIN_TOKEN}", + "var_name": "Authorization", + "location": "header" + } } } ``` -The `scope` field is optional and specifies the level of access that the client is requesting. +## Protocol-Specific Security -## Tool Access Control +### HTTP/HTTPS Security -When exposing tools through UTCP, consider implementing these access controls: - -1. **Tool-Level Permissions**: Define which users/agents can access specific tools -2. **Parameter Constraints**: Restrict parameter values to prevent abuse -3. **Rate Limiting**: Implement per-user/per-tool rate limits -4. **Usage Quotas**: Set maximum usage quotas for tools -5. **Audit Logging**: Log all tool calls for security monitoring +**Required configurations:** +```json +{ + "call_template_type": "http", + "url": "https://api.example.com/endpoint", + "verify_ssl": true, + "timeout": 30, + "headers": { + "User-Agent": "UTCP-Client/1.0", + "X-Request-ID": "${request_id}" + } +} +``` -## Communication Protocol-Specific Considerations +**Security checklist:** +- ✅ Always use HTTPS in production +- ✅ Validate SSL certificates (`verify_ssl: true`) +- ✅ Set appropriate timeouts +- ✅ Include request tracking headers +- ✅ Implement retry limits -### HTTP +### WebSocket Security -- Always use HTTPS, never HTTP +**Secure WebSocket configuration:** +```json +{ + "call_template_type": "websocket", + "url": "wss://api.example.com/ws", + "headers": { + "Authorization": "Bearer ${WS_TOKEN}", + "Origin": "https://trusted-domain.com" + }, + "ping_interval": 30, + "connection_timeout": 10 +} +``` -### CLI +**Security measures:** +- ✅ Use WSS (secure WebSocket) only +- ✅ Validate Origin headers +- ✅ Implement connection timeouts +- ✅ Use heartbeat/ping for connection health +- ✅ Limit concurrent connections per client -:::important +### CLI Security -CLI poses a significant security risk as it executes commands on the local system. +:::danger High Risk Protocol +CLI execution poses significant security risks. Use with extreme caution. ::: -- Validate and sanitize all input parameters -- Run commands with the minimum necessary permissions -- Implement allow-lists for permitted commands -- Sandbox execution environments when possible +**Secure CLI implementation:** +```json +{ + "call_template_type": "cli", + "commands": [ + { + "command": "cd /safe/sandbox", + "append_to_final_output": false + }, + { + "command": "/usr/local/bin/safe-script --input UTCP_ARG_sanitized_input_UTCP_END", + "append_to_final_output": true + } + ], + "working_dir": "/safe/sandbox", + "env_vars": { + "PATH": "/usr/local/bin:/usr/bin", + "HOME": "/tmp/sandbox" + } +} +``` + +**Security requirements:** +- ✅ Use absolute paths for commands +- ✅ Sanitize all input parameters +- ✅ Run in sandboxed environments +- ✅ Limit environment variables +- ✅ Set strict timeouts +- ✅ Validate exit codes +- ✅ Use minimal user permissions + +**Input sanitization requirements:** +- Remove dangerous shell metacharacters: `;`, `&`, `|`, `` ` ``, `$`, `()`, `{}`, `[]`, `<>` +- Escape inputs appropriately for shell execution +- Use parameterized command execution when possible +- Validate inputs against expected patterns +- Implement length limits for all inputs + +### Server-Sent Events (SSE) Security + +**Secure SSE configuration:** +```json +{ + "call_template_type": "sse", + "url": "https://api.example.com/events", + "headers": { + "Authorization": "Bearer ${SSE_TOKEN}", + "Accept": "text/event-stream", + "Cache-Control": "no-cache" + }, + "timeout": 300, + "max_events": 1000 +} +``` + +**Security considerations:** +- ✅ Authenticate SSE connections +- ✅ Set maximum event limits +- ✅ Implement connection timeouts +- ✅ Validate event data format +- ✅ Monitor for event flooding + +### Text Protocol Security + +**Secure file access:** +```json +{ + "call_template_type": "text", + "file_path": "/safe/data/${filename}" +} +``` + +**Security measures:** +- ✅ Restrict file paths to safe directories +- ✅ Set maximum file size limits +- ✅ Validate file extensions +- ✅ Prevent directory traversal attacks +- ✅ Use safe encoding handling + +**Path validation requirements:** +- Resolve all paths to absolute paths +- Check if paths are within allowed directories +- Handle symbolic links by resolving them first +- Validate against directory traversal attacks (`../`) +- Return false for any path resolution errors +- Use allowlists of permitted directories + +### MCP Security + +**Secure MCP server configuration:** +```json +{ + "call_template_type": "mcp", + "config": { + "mcpServers": { + "server_name": { + "transport": "stdio", + "command": ["python", "-m", "my_mcp_server"] + } + } + } +} +``` + +**Security considerations:** +- ✅ Use trusted MCP server implementations +- ✅ Sandbox MCP server processes +- ✅ Limit server resource usage +- ✅ Monitor server health and logs +- ✅ Implement server restart policies + +## Input Validation & Sanitization + +### JSON Schema Validation + +**Comprehensive input validation:** +```json +{ + "inputs": { + "type": "object", + "properties": { + "email": { + "type": "string", + "format": "email", + "maxLength": 254 + }, + "age": { + "type": "integer", + "minimum": 0, + "maximum": 150 + }, + "tags": { + "type": "array", + "items": {"type": "string", "pattern": "^[a-zA-Z0-9_-]+$"}, + "maxItems": 10 + } + }, + "required": ["email"], + "additionalProperties": false + } +} +``` + +### Parameter Sanitization + +**Server-side validation requirements:** +- Use regex patterns to validate input formats (e.g., `^[a-zA-Z0-9_-]+$` for user IDs) +- Implement maximum length limits for all string inputs +- Remove dangerous characters like `<>`, `"`, `'` from user inputs +- Validate all inputs against expected schemas +- Sanitize inputs before processing + +## Secure Variable Handling + +### Environment Variable Security + +**Secure variable loading requirements:** +- Only allow variables with approved prefixes (e.g., `UTCP_`, `API_`) +- Validate variable names against allowlists +- Implement length limits for variable values (e.g., max 10,000 characters) +- Check for dangerous characters in values (`<`, `>`, `"`, `'`) +- Use secure variable substitution methods +- Log all variable access attempts + +### Runtime Variable Substitution + +**Secure substitution requirements:** + +Implement variable substitution with these security measures: +- Only substitute variables matching the pattern `${variable_name}` +- Validate variable names contain only alphanumeric characters and underscores +- Check that all variables exist before substitution +- Sanitize variable values to prevent injection attacks +- Reject values containing dangerous characters like `<`, `>`, `"`, `'`, `;`, `&` +- Limit variable value length to prevent buffer overflow attacks + +## Network & Transport Security + +### TLS Configuration + +**Minimum TLS requirements:** +**TLS configuration requirements:** + +Configure secure HTTP clients with these settings: +- Enable SSL certificate verification +- Set reasonable connection timeouts (e.g., 30 seconds) +- Limit maximum connections to prevent resource exhaustion +- Use TLS 1.2 or higher as minimum version +- Enable hostname verification +- Require certificate verification (CERT_REQUIRED mode) + +### Certificate Validation + +**Enhanced certificate validation:** + +Implement robust certificate validation: +- Use trusted certificate authority bundles +- Enable hostname verification against certificate +- Require valid certificate chains +- Set minimum TLS version to 1.2 +- Configure strong cipher suites (ECDHE+AESGCM, CHACHA20, DHE+AESGCM) +- Reject weak algorithms (aNULL, MD5, DSS) + +## Monitoring & Incident Response + +### Security Logging + +**Comprehensive security logging:** + +Implement security logging with these components: +- **Tool Call Logging**: Record all tool invocations with user ID, tool name, timestamp, success status, and parameters +- **Authentication Logging**: Log authentication attempts, failures, and reasons +- **Structured Format**: Use JSON format for easy parsing and analysis +- **Sensitive Data Protection**: Avoid logging sensitive information like passwords or tokens +- **Audit Trail**: Maintain immutable logs for compliance and forensic analysis + +### Anomaly Detection + +**Basic anomaly detection:** -### WebSocket +Implement anomaly detection with these features: +- **Rate Limiting**: Track requests per user with configurable limits (e.g., 60 calls/minute, 1000 calls/hour) +- **Time Window Management**: Clean old entries and maintain sliding time windows +- **Multi-tier Limits**: Enforce both short-term (per minute) and long-term (per hour) rate limits +- **Automatic Blocking**: Reject requests that exceed configured thresholds +- **Call Tracking**: Record timestamps of all user requests for analysis -- Use secure WebSocket (WSS) connections +## Security Testing & Validation -## Data Protection +### Testing Methodologies -1. **Data in Transit**: Ensure all communications use TLS 1.2+ encryption -2. **Data at Rest**: Encrypt sensitive configuration data -3. **Sensitive Data in Logs**: Prevent logging of sensitive parameters -4. **PII Handling**: Implement proper controls for personal information +**Security test examples:** -## Secure Implementation Checklist +Implement comprehensive security testing: +- **Injection Prevention Tests**: Test for SQL injection, command injection, and other malicious inputs +- **Path Traversal Tests**: Verify protection against directory traversal attacks (../../../etc/passwd) +- **Rate Limiting Tests**: Confirm rate limiting enforcement under high load +- **Authentication Tests**: Validate proper authentication and authorization +- **Input Validation Tests**: Test boundary conditions and malformed inputs -- [ ] Use HTTPS/WSS for all network communications -- [ ] Implement proper authentication for all communication protocols -- [ ] Validate all input against schemas before processing -- [ ] Sanitize inputs to prevent injection attacks -- [ ] Implement rate limiting to prevent abuse -- [ ] Set appropriate timeouts for all operations -- [ ] Log security-relevant events -- [ ] Regularly update dependencies -- [ ] Implement proper error handling that doesn't leak sensitive information +### Security Automation -## Common Vulnerabilities to Avoid +**Automated security checks:** + +Implement automated security validation: +- **Protocol Security**: Verify HTTPS usage instead of HTTP for web requests +- **Credential Detection**: Check for hardcoded passwords, secrets, or API keys +- **Variable Validation**: Ensure proper variable substitution patterns ($\{variable\}) +- **CLI Security**: Validate command-line tools use absolute paths and safe commands +- **URL Validation**: Check for suspicious or malformed URLs +- **Configuration Review**: Automated scanning of UTCP manuals for security issues -| Vulnerability | Prevention | -|--------------|------------| -| Injection Attacks | Validate and sanitize all inputs | -| Credential Leakage | Use secure credential storage | -| Excessive Permissions | Follow the principle of least privilege | -| Man-in-the-Middle | Use certificate validation and pinning | -| Denial of Service | Implement rate limiting and timeouts | -| Information Disclosure | Ensure errors don't leak sensitive data | +## Security Checklist -## Secure Development Lifecycle +### Tool Provider Security + +- [ ] UTCP manual endpoint requires authentication +- [ ] All tool endpoints use HTTPS/WSS +- [ ] Input validation implemented for all tools +- [ ] Rate limiting configured per user/tool +- [ ] Security logging enabled +- [ ] Credentials stored securely (not hardcoded) +- [ ] SSL certificate validation enabled +- [ ] Appropriate timeouts configured +- [ ] Error messages don't leak sensitive information -1. **Design**: Conduct threat modeling during protocol design -2. **Implementation**: Follow secure coding practices -3. **Testing**: Perform security testing and code reviews -4. **Deployment**: Use secure deployment practices -5. **Maintenance**: Monitor for security issues and update regularly +### Tool Consumer Security + +- [ ] Variable substitution is sanitized +- [ ] SSL certificate verification enabled +- [ ] Connection timeouts configured +- [ ] Rate limiting respected +- [ ] Security events logged +- [ ] Credentials rotated regularly +- [ ] Network connections monitored +- [ ] Input validation before tool calls -By following these security considerations, UTCP implementations can minimize risks while enabling powerful tool integrations across various communication protocols. +### Protocol-Specific Security + +- [ ] **HTTP**: HTTPS only, certificate validation +- [ ] **WebSocket**: WSS only, origin validation +- [ ] **CLI**: Sandboxed execution, input sanitization +- [ ] **SSE**: Authenticated connections, event limits +- [ ] **Text**: Path validation, size limits +- [ ] **MCP**: Trusted servers, resource limits + +By following these security guidelines, you can safely implement UTCP while maintaining strong security posture across all communication protocols. + +For protocol-specific security details, see: +- [HTTP Security](./protocols/http.md#security-considerations) +- [WebSocket Security](./protocols/websocket.md#security-considerations) +- [CLI Security](./protocols/cli.md#security-considerations) +- [SSE Security](./protocols/sse.md#security-considerations) +- [Text Security](./protocols/text.md#security-considerations) +- [MCP Security](./protocols/mcp.md#security-considerations) diff --git a/versioned_docs/version-1.0/utcp-vs-mcp.md b/versioned_docs/version-1.0/utcp-vs-mcp.md index 3dec676..bc670ff 100644 --- a/versioned_docs/version-1.0/utcp-vs-mcp.md +++ b/versioned_docs/version-1.0/utcp-vs-mcp.md @@ -4,99 +4,478 @@ title: UTCP vs MCP sidebar_position: 5 --- -# UTCP vs MCP tool calling: A Comparison +# UTCP vs MCP: A Comprehensive Comparison -This page compares the Universal Tool Calling Protocol (UTCP) with the Model Context Protocol's (MCP) tool calling functionality, highlighting their different approaches to agent-tool integration. +:::info Language Examples +This comparison uses **Python** examples. Both UTCP and MCP have implementations in multiple languages - check respective GitHub organizations for language-specific examples. +::: + +This guide compares the Universal Tool Calling Protocol (UTCP) with the Model Context Protocol (MCP), helping you choose the right approach for your AI tool integration needs. ## Video Overview -## Architectural Differences - -| Aspect | MCP | UTCP | -|--------|-----|------| -| **Core Model** | Middleman | Manual | -| **Architecture** | Agents ↔ MCP Server ↔ Tool | Agent ↔ Tool (Direct) | -| **Integration Approach** | Wraps existing tools | Describes how to call existing tools | -| **Network Hops** | Double (Agent → MCP → Tool) | Single (Agent → Tool) | -| **Protocol Dependency** | Hard dependency on protocol for every call | Protocol only needed during discovery | +## Quick Comparison -## The Middleman vs Manual Philosophies +| Aspect | UTCP | MCP | +|--------|------|-----| +| **Philosophy** | Manual (describes how to call tools) | Middleman (wraps tools in protocol) | +| **Architecture** | Agent → Tool (Direct) | Agent → MCP Server → Tool | +| **Infrastructure** | None required | Wrapper servers needed | +| **Protocols** | HTTP, WebSocket, CLI, SSE, etc. | JSON-RPC over stdio/HTTP | +| **Performance** | Native tool performance | Additional proxy overhead | +| **Maintenance** | Minimal | High (server maintenance) | -### The MCP "Middleman" Approach +## Architectural Differences -MCP positions itself as the "USB-C for AI Agents" — a universal adapter that all tools must plug into. This approach: +### UTCP: The "Manual" Approach + +UTCP provides a standardized way to describe how to call existing tools directly: + +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [{ + "name": "get_weather", + "description": "Get current weather", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.weather.com/current", + "http_method": "GET", + "query_params": {"location": "${location}"} + } + }] +} +``` + +**Flow:** Agent discovers manual → Agent calls tool directly + +### MCP: The "Middleman" Approach + +MCP requires building servers that wrap your tools: + +**MCP Server Implementation Requirements:** +- Create a dedicated server process for each tool provider +- Implement tool listing functionality to expose available tools +- Implement tool calling handlers that proxy requests to actual APIs +- Maintain server infrastructure and handle client connections +- Route all tool calls through the MCP server layer + +**Flow:** Agent → MCP Server → Tool → MCP Server → Agent + +## Technical Comparison + +### Performance Impact + +#### UTCP Performance + +**Direct API calls with no overhead:** +- Make HTTP requests directly to weather service endpoints +- No intermediate proxy servers or additional network hops +- Latency equals API response time only (~100ms) +- Native HTTP client performance with connection pooling + +#### MCP Performance + +**Requires MCP server proxy:** +- Connect to MCP server before making tool calls +- Route requests through MCP server to actual weather API +- Additional network hop adds latency overhead +- Latency includes API response time plus MCP server processing (~150ms) + +### Infrastructure Requirements + +#### UTCP Infrastructure + +**Minimal infrastructure requirements:** +- Add single discovery endpoint to existing API (e.g., GET /utcp) +- Return static JSON manual describing available tools +- No additional servers, processes, or infrastructure needed +- Total infrastructure: 0 additional servers + +#### MCP Infrastructure + +**MCP infrastructure requirements:** +- Requires dedicated MCP server processes for each tool provider +- Process management, monitoring, and scaling infrastructure needed +- Client connection management and session handling required +- Total infrastructure: N MCP servers (one per tool provider) + +### Protocol Support Comparison + +#### UTCP Protocol Flexibility +```json +{ + "tools": [ + { + "name": "http_tool", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/data" + } + }, + { + "name": "websocket_tool", + "tool_call_template": { + "call_template_type": "websocket", + "url": "wss://api.example.com/ws" + } + }, + { + "name": "cli_tool", + "tool_call_template": { + "call_template_type": "cli", + "commands": [ + { + "command": "git status --porcelain", + "append_to_final_output": true + } + ] + } + } + ] +} +``` + +#### MCP Protocol Limitation + +**MCP protocol constraints:** +- Only supports JSON-RPC over stdio/HTTP transport +- All tools must be wrapped in MCP server implementations +- Cannot directly call WebSocket, CLI, or other native protocols +- Requires protocol translation layer for non-HTTP tools -- Forces all traffic through a new protocol layer -- Requires writing "wrappers" for existing tools -- Needs to reinvent solutions for auth, security, and other infrastructure concerns +## Feature Comparison -This creates what we call the "wrapper tax": the additional infrastructure, development, and maintenance overhead required to adapt existing tools to work with MCP. +### Authentication & Security + +#### UTCP: Native Authentication +```json +{ + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.example.com/data", + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token" + } + } +} +``` + +**Benefits:** +- Uses existing authentication systems +- No credential translation needed +- Native rate limiting and monitoring +- Existing security policies apply + +#### MCP: Server-Mediated Authentication + +**MCP server authentication requirements:** +- MCP server must handle authentication translation between client and API +- Server stores and manages API credentials on behalf of clients +- Server makes authenticated calls to actual APIs using stored credentials +- Requires credential management and secure storage in MCP server + +**Challenges:** +- Credential management in MCP servers +- Additional security layer to maintain +- Auth translation complexity + +### Streaming & Real-time Data + +#### UTCP: Native Streaming Support +```json +{ + "name": "stream_logs", + "tool_call_template": { + "call_template_type": "sse", + "url": "https://api.example.com/logs/stream", + "timeout": 300 + } +} +``` + +#### MCP: Limited Streaming + +**MCP streaming limitations:** +- MCP has basic streaming capabilities but requires server implementation +- More complex to set up and maintain than native streaming protocols +- Additional abstraction layer between client and streaming data source + +### Error Handling + +#### UTCP: Native Error Responses + +**Direct error handling:** +- Errors come directly from the tool without translation +- Native HTTP status codes and error messages preserved +- Full error context available including headers and response body +- No error translation or abstraction layer + +#### MCP: Wrapped Error Responses + +**Error abstraction layer:** +- Errors are wrapped and translated by MCP server +- Original error context may be lost in translation +- MCP-specific error format instead of native tool errors +- Additional debugging complexity due to error wrapping + +## Migration & Interoperability + +### Migrating from MCP to UTCP + +UTCP provides an MCP plugin for gradual migration: + +**Migration Strategy:** +- **Phase 1**: Use existing MCP servers via UTCP's MCP protocol plugin +- Configure UTCP client to connect to legacy MCP servers using MCP call templates +- **Phase 2**: Migrate high-value tools to native UTCP protocols (HTTP, WebSocket, CLI) +- **Phase 3**: Deprecate MCP servers once migration is complete + +[**Complete migration guide →**](./protocols/mcp.md) + +### Hybrid Approach + +You can use both protocols simultaneously: + +**Hybrid approach during migration:** +- Configure UTCP client with both native UTCP and legacy MCP call templates +- Native UTCP tools use direct HTTP/WebSocket/CLI protocols +- Legacy MCP tools continue using MCP protocol plugin +- Gradually migrate tools from MCP to native UTCP protocols +- Single client interface for both native and legacy tools + +## Enterprise Decision Factors + +### Total Cost of Ownership + +#### UTCP TCO +``` +Infrastructure: $0 (uses existing APIs) +Development: Low (add one endpoint) +Maintenance: Minimal (static JSON) +Scaling: Automatic (scales with existing API) +Monitoring: Existing tools work +``` + +#### MCP TCO +``` +Infrastructure: High (dedicated servers) +Development: High (build wrapper servers) +Maintenance: High (server management) +Scaling: Complex (scale MCP servers separately) +Monitoring: Additional monitoring stack needed +``` -### The UTCP "Manual" Approach +### Development Velocity -UTCP takes a different approach — it's a "manual" that describes how to call tools directly: +#### UTCP Development Speed + +**Rapid deployment timeline:** +- **Day 1**: Add UTCP discovery endpoint to existing API +- **Day 2**: Tools are immediately available to AI agents +- No additional infrastructure, servers, or deployment needed +- Minimal code changes to existing systems -- Provides all necessary information to call native APIs directly -- Gets out of the way after tool discovery -- Leverages existing infrastructure for auth, security, etc. +#### MCP Development Speed -This eliminates the wrapper tax and allows organizations to expose their existing APIs to AI agents without building and maintaining additional infrastructure. +**Extended development timeline:** +- **Week 1-2**: Build dedicated MCP server implementation +- **Week 3**: Deploy and configure server infrastructure +- **Week 4**: Set up monitoring, logging, and scaling +- **Week 5**: Handle production issues and debugging +- **Ongoing**: Server maintenance, updates, and operations -## Feature Comparison +### Risk Assessment -| Feature | MCP | UTCP | -|---------|-----|------| -| **Tool Discovery** | Via MCP Server | Via manual discovery endpoint | -| **Protocol Support** | HTTP Streaming | HTTP, WebSockets, gRPC, CLI, etc. | -| **Authentication** | Handled by MCP Server | Uses tool's native authentication | -| **Streaming** | Native support | Supported via appropriate communication protocol (SSE, WebSockets) | -| **Implementation Complexity** | High (requires wrapper servers) | Low (simple JSON definitions) | -| **Performance** | Additional overhead due to proxy | Direct, native performance | -| **Evolution Speed** | Slow (all participants must update) | Fast (individual communication protocols can evolve independently) | +| Risk Factor | UTCP | MCP | +|-------------|------|-----| +| **Single Point of Failure** | None (direct calls) | MCP servers | +| **Vendor Lock-in** | Low (standard protocols) | Medium (MCP-specific) | +| **Maintenance Burden** | Low | High | +| **Security Surface** | Minimal | Expanded | +| **Performance Risk** | Low | Medium | -## When to Choose Each Protocol - -### Choose MCP When: - -- You need strict standardization across all tools -- You're building a closed ecosystem where you control all components -- You're willing to invest in building and maintaining wrapper servers +## Decision Framework ### Choose UTCP When: -- You want to leverage existing APIs without building wrappers -- You need to support diverse communication protocols -- You value direct, efficient communication -- You prioritize low implementation overhead -- You want to minimize infrastructure costs +✅ **You have existing APIs** that work well +✅ **You want minimal infrastructure** overhead +✅ **You need multiple protocols** (HTTP, WebSocket, CLI, etc.) +✅ **You prioritize performance** and direct communication +✅ **You want to leverage existing** auth, monitoring, scaling +✅ **You have limited resources** for server maintenance +✅ **You need rapid deployment** of AI tool access + +### Choose MCP When: -## Real-World Example +✅ **You need strict protocol standardization** across all tools +✅ **You're building a closed ecosystem** with full control +✅ **You have resources** for building and maintaining servers +✅ **You need MCP-specific features** like resources and prompts +✅ **You're already invested** in MCP infrastructure +✅ **You prefer centralized control** over tool access + +### Hybrid Approach When: + +✅ **You're migrating from MCP** to UTCP gradually +✅ **You have mixed requirements** (some tools need MCP features) +✅ **You want to evaluate both** approaches in production +✅ **You have legacy MCP investments** to preserve + +## Real-World Examples + +### E-commerce API Integration + +#### UTCP Approach + +**E-commerce API with UTCP:** +- Keep existing product API endpoints unchanged (GET /products/\{product_id\}) +- Add single UTCP discovery endpoint (GET /utcp) +- Return UTCP manual describing available tools and how to call them +- Tools directly reference existing API endpoints with proper parameters +- Total additional code: ~10 lines +- Additional infrastructure: 0 servers + +#### MCP Approach + +**E-commerce API with MCP:** +- Requires building dedicated MCP server wrapper +- Implement tool listing and calling handlers in MCP server +- MCP server calls existing API endpoints on behalf of clients +- Additional server deployment, monitoring, and scaling required +- Total additional code: ~50+ lines +- Additional infrastructure: 1+ servers + +### Database Query Tool + +#### UTCP Approach +```json +{ + "name": "query_database", + "tool_call_template": { + "call_template_type": "http", + "url": "https://api.company.com/query", + "http_method": "POST", + "body": {"sql": "${query}"}, + "auth": { + "auth_type": "api_key", + "api_key": "${DB_API_KEY}", + "var_name": "Authorization", + "location": "header" + } + } +} +``` + +#### MCP Approach +**MCP database approach:** +- Requires MCP server with database connection management +- Connection pooling, query validation, and security in MCP server +- Much more complex implementation than direct database access +- Additional abstraction layer between client and database + +## Performance Benchmarks + +### Latency Comparison + +| Scenario | UTCP | MCP | Difference | +|----------|------|-----|------------| +| Simple API call | 50ms | 75ms | +50% overhead | +| Complex query | 200ms | 250ms | +25% overhead | +| File operation | 10ms | 20ms | +100% overhead | +| Streaming data | Real-time | Buffered | Significant delay | + +### Resource Usage + +| Resource | UTCP | MCP | +|----------|------|-----| +| Memory | 0MB (no servers) | 50-200MB per server | +| CPU | 0% (no processing) | 5-15% per server | +| Network | Direct | Double hops | +| Storage | 0GB | Logs, state, config | + +## Migration Timeline + +### From MCP to UTCP + +**Phase 1 (Week 1): Assessment** +- Inventory existing MCP servers +- Identify high-value tools for migration +- Plan migration strategy + +**Phase 2 (Week 2-3): Hybrid Setup** +- Install UTCP with MCP plugin +- Test existing MCP tools through UTCP +- Validate functionality + +**Phase 3 (Week 4-8): Gradual Migration** +- Migrate tools one by one to native UTCP +- Add `/utcp` endpoints to existing APIs +- Update client configurations + +**Phase 4 (Week 9+): Cleanup** +- Deprecate MCP servers +- Remove MCP infrastructure +- Monitor and optimize + +[**Detailed migration guide →**](./migration-v0.1-to-v1.0.md) + +## Community & Ecosystem + +### UTCP Ecosystem +- **Multiple language implementations**: Python, TypeScript, Go, Rust +- **Growing protocol support**: HTTP, WebSocket, CLI, SSE, Text, MCP +- **Active development**: Regular releases and improvements +- **Open governance**: RFC process for changes + +### MCP Ecosystem +- **Anthropic-led development**: Centralized development +- **Growing tool library**: Community-contributed servers +- **IDE integrations**: Claude Desktop, Cline, etc. +- **Established patterns**: Well-documented server patterns -Consider an organization with an existing REST API that they want to expose to AI agents: +## Conclusion -**With MCP:** -1. Build an MCP server that wraps the REST API -2. Translate all calls between MCP format and REST format -3. Maintain and scale this additional server infrastructure -4. Handle authentication translation between MCP and the API +Both UTCP and MCP solve the tool integration problem, but with fundamentally different approaches: -**With UTCP:** -1. Create a simple JSON definition describing the REST API -2. Expose this definition via a discovery endpoint (typically `/utcp`) -3. The AI agent can now call the REST API directly +**UTCP excels when you:** +- Want to leverage existing APIs without additional infrastructure +- Need support for multiple communication protocols +- Prioritize performance and direct communication +- Have limited resources for server maintenance +- Want rapid deployment and minimal complexity -## Code comparison +**MCP excels when you:** +- Need strict protocol standardization +- Are building a controlled ecosystem +- Have resources for server infrastructure +- Need MCP-specific features beyond tool calling +- Prefer centralized tool management -You can find a full typescript example detailing the MCP and UTCP approach [here](https://github.com/universal-tool-calling-protocol/typescript-utcp/tree/main/examples/src/concrete_example). +**For most organizations**, UTCP's "manual" approach offers significant advantages in terms of simplicity, performance, and cost-effectiveness. The ability to expose existing APIs to AI agents with minimal changes and no additional infrastructure makes it an attractive choice for rapid AI tool integration. -## Conclusion +**For gradual adoption**, consider starting with UTCP's MCP plugin to use existing MCP servers while migrating high-value tools to native UTCP protocols over time. -Both MCP and UTCP aim to solve the problem of standardizing tool calling for AI agents, but they take fundamentally different approaches. +## Next Steps -MCP acts as a middleman, requiring all tools to be wrapped in its protocol. This provides standardization but at the cost of additional infrastructure and development overhead. +### To Get Started with UTCP: +1. **[Read the implementation guide](./implementation.md)** - Learn how to implement UTCP +2. **[Choose your protocols](./protocols/index.md)** - Select communication methods +3. **[Check examples](https://github.com/universal-tool-calling-protocol)** - See real implementations across multiple languages -UTCP acts as a manual, describing how to call tools directly using their native interfaces. This eliminates the wrapper tax and leverages existing infrastructure, at the cost of requiring clients to handle different communication protocols. +### To Migrate from MCP: +1. **[Read the MCP integration guide](./protocols/mcp.md)** - Use MCP tools via UTCP +2. **[Plan your migration](./migration-v0.1-to-v1.0.md)** - Step-by-step migration process +3. **[Join the community](https://discord.gg/ZpMbQ8jRbD)** - Get migration support -The choice between them depends on your specific requirements, existing infrastructure, and development resources. However, UTCP's "manual" approach offers significant advantages in terms of simplicity, efficiency, and leveraging existing investments in API infrastructure. +### To Learn More: +- **[UTCP Architecture](./api/index.md)** - Technical deep dive +- **[Security Considerations](./security.md)** - Security best practices +- **[Tool Provider Guide](./for-tool-providers.md)** - Expose your tools