diff --git a/.github/workflows/build-python.yaml b/.github/workflows/build-python.yaml index 28a4a8e..3a0af33 100644 --- a/.github/workflows/build-python.yaml +++ b/.github/workflows/build-python.yaml @@ -1,9 +1,6 @@ # Build otdf-python wheel using uv and output the wheel path for downstream workflows name: "Build Python Wheel" on: - push: - branches: - - chore/rewrite pull_request: jobs: diff --git a/README.md b/README.md index 9b63337..60be913 100644 --- a/README.md +++ b/README.md @@ -83,8 +83,6 @@ with open("encrypted.tdf", "wb") as f: ### Decrypt Data ```python -from otdf_python.tdf import TDFReaderConfig - # Read encrypted TDF file with open("encrypted.tdf", "rb") as f: encrypted_data = f.read() diff --git a/docs/CONNECT_RPC.md b/docs/CONNECT_RPC.md new file mode 100644 index 0000000..eb37da2 --- /dev/null +++ b/docs/CONNECT_RPC.md @@ -0,0 +1,156 @@ +# Connect RPC in OpenTDF Python SDK + +This document describes the Connect RPC implementation in the OpenTDF Python SDK, which provides a modern HTTP-friendly alternative to traditional gRPC. + +## Overview + +Connect RPC is a protocol that brings the benefits of RPC to HTTP APIs. It's designed to be: +- **HTTP-compatible**: Works with standard HTTP infrastructure +- **Type-safe**: Generated from Protocol Buffer definitions +- **Efficient**: Binary protocol with JSON fallback +- **Simple**: Easy to debug and integrate + +## Architecture + +The SDK uses a two-module structure for protocol buffer generation: + +- **Main SDK**: `src/otdf_python/` - Core SDK functionality +- **Protocol Generation**: `otdf-python-proto/` - Generated Connect RPC and protobuf files + +### Generated Files + +Connect RPC generates the following files in `otdf-python-proto/src/otdf_python_proto/`: + +- `*_connect.py` - Connect RPC client implementations (recommended) +- `*_pb2.py` - Protocol buffer message definitions +- `legacy_grpc/*_pb2_grpc.py` - Traditional gRPC clients (backward compatibility) + +## Usage + +### Client Creation + +Connect RPC clients are created using the generated `*_connect.py` modules: + +```python +from otdf_python_proto.kas import kas_connect +from otdf_python_proto.policy import policy_connect + +# Create Connect RPC clients +kas_client = kas_connect.KeyAccessServiceClient(base_url="https://example.com") +policy_client = policy_connect.PolicyServiceClient(base_url="https://your-policy-endpoint") +``` + +### Authentication + +Connect RPC clients support standard HTTP authentication: + +```python +import httpx + +# Create authenticated HTTP client, assuming `token` holds your auth token +http_client = httpx.Client( + headers={"Authorization": f"Bearer {token}"} +) + +# Use with Connect RPC client +kas_client = kas_connect.KeyAccessServiceClient( + base_url="https://example.com", + http_client=http_client +) +``` + +## Regenerating Connect RPC Files + +To regenerate the Connect RPC and protobuf files: + +```bash +cd otdf-python-proto +uv run python scripts/generate_connect_proto.py +``` + +### Download Fresh Proto Files + +To download the latest protocol definitions: + +```bash +cd otdf-python-proto +uv run python scripts/generate_connect_proto.py --download +``` + +### Requirements + +- `buf` tool: `brew install bufbuild/buf/buf` (or see [official installation guide](https://buf.build/docs/installation)) +- Python dependencies managed by `uv` + +## Benefits Over gRPC + +1. **HTTP Compatibility**: Works with load balancers, proxies, and other HTTP infrastructure +2. **Debugging**: Easy to inspect with standard HTTP tools +3. **Flexibility**: Supports both binary and JSON encoding +4. **Simplicity**: No special HTTP/2 requirements + +## Migration from gRPC + +If migrating from legacy gRPC clients: + +1. Replace `*_pb2_grpc.py` imports with `*_connect.py` +2. Update client instantiation to use base URLs instead of channels +3. Leverage HTTP client features for authentication and configuration + +## Testing + +Connect RPC clients can be easily mocked and tested using standard HTTP testing tools: + +```python +import httpx +import respx +from otdf_python_proto.kas import kas_pb2, kas_connect + +@respx.mock +def test_connect_rpc_client(): + # 1. Create a sample protobuf response message + expected_response = kas_pb2.RewrapResponse( + entity_wrapped_key=b'some-unwrapped-key' + ) + + # 2. Mock the correct Connect RPC endpoint URL + respx.post("https://example.com/kas.AccessService/Rewrap").mock( + return_value=httpx.Response( + 200, + # 3. Return the serialized protobuf message as content + content=expected_response.SerializeToString(), + headers={'Content-Type': 'application/proto'} + ) + ) + + client = kas_connect.KeyAccessServiceClient(base_url="https://example.com") + # Test client calls... +``` + +## Error Handling + +Connect RPC provides structured error handling through standard HTTP status codes and error details: + +```python +from connectrpc import ConnectError + +try: + # Assuming `kas_client` and `request` are defined as in previous examples + response = kas_client.rewrap(request) +except ConnectError as e: + print(f"Connect RPC error: {e.code} - {e.message}") + # Handle specific error types +``` + +## Performance Considerations + +- Use connection pooling with `httpx.Client` for better performance +- Configure appropriate timeouts for your use case +- Consider using binary encoding for high-throughput scenarios + +## Additional Resources + +For more information, see: +- [Connect RPC Documentation](https://connectrpc.com/docs/) +- [Connect Python Repository](https://github.com/connectrpc/connect-python) +- [OpenTDF Platform](https://github.com/opentdf/platform) diff --git a/docs/CONNECT_RPC_MIGRATION.md b/docs/CONNECT_RPC_MIGRATION.md deleted file mode 100644 index 92ef6fc..0000000 --- a/docs/CONNECT_RPC_MIGRATION.md +++ /dev/null @@ -1,283 +0,0 @@ -# Connect RPC Migration Guide - -This document explains how to migrate from traditional gRPC clients to Connect RPC clients in the OpenTDF Python SDK. - -## What is Connect RPC? - -Connect RPC is a modern, HTTP-friendly alternative to gRPC that provides: - -- **HTTP/1.1 compatibility** - Works with all HTTP infrastructure -- **Human-readable debugging** - JSON payloads can be inspected with standard tools -- **Browser compatibility** - Can be called directly from web browsers -- **Simplified deployment** - No special gRPC infrastructure required -- **Better observability** - Standard HTTP status codes and headers - -For more information, see the [Connect RPC Protocol Documentation](https://connectrpc.com/docs/protocol/). - -## Dependencies - -The project now includes both Connect RPC and legacy gRPC dependencies: - -```toml -dependencies = [ - "connect-python>=0.4.2", # Connect RPC client - "grpcio>=1.74.0", # Legacy gRPC (backward compatibility) - "grpcio-tools>=1.74.0", # Legacy gRPC tools - # ... other dependencies -] -``` - -## Code Generation - -### Connect RPC Generation (Recommended) - -Use the new Connect RPC generation script: - -```bash -cd proto-gen -uv run python scripts/generate_connect_proto.py -``` - -This generates: -- `*_connect.py` - Connect RPC clients (preferred) -- `*_pb2.py` - Standard protobuf classes -- `*_pb2.pyi` - Type stubs -- `legacy_grpc/*_pb2_grpc.py` - Legacy gRPC clients (backward compatibility) - -### Legacy gRPC Generation - -The old script still works for backward compatibility: - -```bash -cd proto-gen -uv run python scripts/generate_proto.py -``` - -## Client Usage Examples - -### Connect RPC Client (Recommended) - -```python -import urllib3 -from otdf_python_proto.policy_pb2 import GetPolicyRequest -from otdf_python_proto.policy_connect import PolicyServiceClient - -# Create HTTP client -http_client = urllib3.PoolManager() - -# Create Connect RPC client -policy_client = PolicyServiceClient( - base_url="https://platform.opentdf.io", - http_client=http_client -) - -# Make unary RPC call -request = GetPolicyRequest(id="policy-123") -response = policy_client.get_policy(request) -print(f"Policy: {response}") - -# With extra headers and timeout -response = policy_client.get_policy( - request, - extra_headers={"Authorization": "Bearer your-token"}, - timeout_seconds=30.0 -) -``` - -### Async Connect RPC Client - -```python -import aiohttp -from otdf_python_proto.policy_pb2 import ListPoliciesRequest -from otdf_python_proto.policy_connect import AsyncPolicyServiceClient - -async def main(): - async with aiohttp.ClientSession() as http_client: - policy_client = AsyncPolicyServiceClient( - base_url="https://platform.opentdf.io", - http_client=http_client - ) - - # Make async RPC call - request = ListPoliciesRequest() - response = await policy_client.list_policies(request) - print(f"Policies: {response}") - - # Server streaming example - async for policy in policy_client.stream_policies(request): - print(f"Streaming policy: {policy}") -``` - -### Legacy gRPC Client (Backward Compatibility) - -```python -import grpc -from otdf_python_proto.policy_pb2 import GetPolicyRequest -from otdf_python_proto.legacy_grpc.policy_pb2_grpc import PolicyServiceStub - -# Create gRPC channel -channel = grpc.insecure_channel("platform.opentdf.io:443") -policy_client = PolicyServiceStub(channel) - -# Make RPC call -request = GetPolicyRequest(id="policy-123") -response = policy_client.GetPolicy(request) -print(f"Policy: {response}") -``` - -## Error Handling - -### Connect RPC Error Handling - -```python -from connectrpc.errors import ConnectError - -try: - response = policy_client.get_policy(request) -except ConnectError as e: - print(f"Connect error: {e.code} - {e.message}") - # e.code can be: "not_found", "permission_denied", etc. - # Full list: https://connectrpc.com/docs/protocol/#error-codes -``` - -### gRPC Error Handling - -```python -import grpc - -try: - response = policy_client.GetPolicy(request) -except grpc.RpcError as e: - print(f"gRPC error: {e.code()} - {e.details()}") -``` - -## Protocol Differences - -| Feature | Connect RPC | gRPC | -|---------|-------------|------| -| Transport | HTTP/1.1, HTTP/2 | HTTP/2 only | -| Payload | JSON or Binary | Binary only | -| Status Codes | HTTP status codes | gRPC status codes | -| Headers | Standard HTTP headers | Custom gRPC headers | -| Browser Support | ✅ Yes | ❌ No (requires gRPC-Web) | -| Debugging | ✅ Human-readable | ❌ Binary format | -| Infrastructure | ✅ Standard HTTP | ❌ Requires gRPC support | - -## Migration Checklist - -- [ ] Update dependencies to include `connect-python` -- [ ] Regenerate proto files with Connect RPC support -- [ ] Update client code to use Connect RPC clients -- [ ] Update error handling for Connect error types -- [ ] Test with your authentication/authorization setup -- [ ] Update deployment configuration (if needed) -- [ ] Remove legacy gRPC dependencies (optional) - -## Advanced Usage - -### Custom HTTP Configuration - -```python -import urllib3 - -# Configure HTTP client with custom settings -http_client = urllib3.PoolManager( - timeout=urllib3.Timeout(connect=10.0, read=30.0), - retries=urllib3.Retry(total=3, backoff_factor=0.3), - headers={"User-Agent": "MyApp/1.0"} -) - -policy_client = PolicyServiceClient( - base_url="https://platform.opentdf.io", - http_client=http_client -) -``` - -### Low-level API Access - -```python -# Access response metadata -output = policy_client.call_get_policy(request) -response = output.message() -headers = output.response_headers() -trailers = output.response_trailers() - -if output.error(): - raise output.error() -``` - -### Server Streaming - -```python -# Server streaming RPC -request = StreamPoliciesRequest() -for policy in policy_client.stream_policies(request): - print(f"Received policy: {policy.id}") - -# With error handling -try: - for policy in policy_client.stream_policies(request): - process_policy(policy) -except ConnectError as e: - print(f"Stream error: {e.code} - {e.message}") -``` - -## Troubleshooting - -### Common Issues - -1. **"buf command not found"** - ```bash - # Install buf - brew install bufbuild/buf/buf - # Or - go install github.com/bufbuild/buf/cmd/buf@latest - ``` - -2. **"protoc-gen-connect_python not found"** - ```bash - # Install with compiler support - uv add connect-python[compiler] - ``` - -3. **Import errors after generation** - ```bash - # Ensure __init__.py files exist - find proto-gen/generated -type d -exec touch {}/__init__.py \; - ``` - -4. **HTTP/2 server issues** - - Connect RPC works with HTTP/1.1, so this is rarely an issue - - If using streaming, ensure your server supports Connect protocol - -### Debug HTTP Traffic - -```python -import logging - -# Enable HTTP debug logging -logging.basicConfig(level=logging.DEBUG) -urllib3.disable_warnings() - -# You can now see all HTTP requests/responses -``` - -## Performance Considerations - -- **HTTP/1.1**: Good for most use cases, supports connection pooling -- **JSON vs Binary**: Binary protobuf is more efficient, JSON is more debuggable -- **Connection Reuse**: Reuse `urllib3.PoolManager` instances across calls -- **Async**: Use async clients for high-concurrency applications - -## Next Steps - -1. **Start with unary RPCs**: Easiest to migrate and test -2. **Test authentication**: Ensure your auth tokens work with HTTP headers -3. **Migrate streaming RPCs**: More complex but follow similar patterns -4. **Remove gRPC dependencies**: Once fully migrated, clean up dependencies -5. **Update documentation**: Update your team's documentation and examples - -For more information, see: -- [Connect RPC Documentation](https://connectrpc.com/docs/) -- [Connect Python Repository](https://github.com/connectrpc/connect-python) -- [OpenTDF Platform](https://github.com/opentdf/platform) diff --git a/docs/PROTOBUF_SETUP.md b/docs/PROTOBUF_SETUP.md deleted file mode 100644 index 053d061..0000000 --- a/docs/PROTOBUF_SETUP.md +++ /dev/null @@ -1,135 +0,0 @@ -# OpenTDF Python SDK - Protobuf Generation Sub-Module - -This document explains the protobuf generation sub-module that was created for the OpenTDF Python SDK project. - -## Overview - -A dedicated sub-module (`proto-gen/`) has been created to handle downloading and generating protobuf files from the OpenTDF platform. This provides a clean separation of concerns and makes it easy to update protobuf definitions. - -## Structure - -``` -opentdf-python-sdk.rewrite/ -├── proto-gen/ # Protobuf generation sub-module -│ ├── pyproject.toml # Sub-module dependencies -│ ├── README.md # Sub-module documentation -│ ├── proto-files/ # Raw .proto files -│ │ ├── kas.proto # Downloaded from OpenTDF platform -│ │ └── kas_simplified.proto # Simplified version (auto-generated) -│ ├── generated/ # Generated Python files -│ │ ├── __init__.py -│ │ ├── kas_pb2.py # Generated protobuf classes -│ │ └── kas_pb2_grpc.py # Generated gRPC service stubs -│ └── scripts/ -│ ├── generate_proto.py # Python script to generate protobuf files -│ └── build_proto.sh # Shell script wrapper -├── scripts/ -│ └── update-proto.sh # Convenience script to regenerate and sync -├── uv.toml # UV workspace configuration -└── pyproject.toml # Main project (includes proto dependency) -``` - -## What Was Accomplished - -### 1. Downloaded Proto File ✅ -- Downloaded the latest `kas.proto` file from: `https://raw.githubusercontent.com/opentdf/platform/refs/tags/service/v0.8.0/service/kas/kas.proto` -- Stored in `proto-gen/proto-files/kas.proto` - -### 2. Built Usable Library ✅ -- Created a robust protobuf generation system using `uv run python -m grpc_tools.protoc` -- Handles dependency issues gracefully with fallback generation -- Generates both protobuf classes (`kas_pb2.py`) and gRPC service stubs (`kas_pb2_grpc.py`) - -## Key Features - -### Smart Dependency Handling -- Automatically detects and uses `googleapis-common-protos` when available -- Falls back to a simplified proto definition when external dependencies are missing -- Handles import issues gracefully - -### Multiple Ways to Generate -1. **Python script**: `uv run python scripts/generate_proto.py` -2. **Shell script**: `./scripts/build_proto.sh` -3. **Convenience script**: `./scripts/update-proto.sh` (from main project) - -### Workspace Integration -- Uses UV workspace configuration to link the sub-module -- Main project depends on `otdf-python-proto` for the generated files -- Automatic syncing of generated files to the main project's proto directory - -## Usage - -### Regenerate Protobuf Files - -From the main project root: -```bash -./scripts/update-proto.sh -``` - -From the proto-gen sub-module: -```bash -cd proto-gen -uv run python scripts/generate_proto.py -``` - -### Update Proto Definition - -1. Download the latest proto file: -```bash -curl -o proto-gen/proto-files/kas.proto https://raw.githubusercontent.com/opentdf/platform/refs/tags/service/v0.8.0/service/kas/kas.proto -``` - -2. Regenerate the Python files: -```bash -./scripts/update-proto.sh -``` - -## Dependencies - -The otdf-python-proto sub-module includes these dependencies: -- `grpcio>=1.74.0` - gRPC runtime -- `grpcio-tools>=1.74.0` - Protocol buffer compiler -- `protobuf>=6.31.1` - Protocol buffer runtime -- `googleapis-common-protos>=1.66.0` - Google API common proto definitions - -## Final Status ✅ - -The protobuf sub-module has been successfully implemented and tested: - -### ✅ Completed Tasks -1. **Downloaded proto file** from OpenTDF platform (service/v0.8.0) -2. **Built usable library** with `uv run python -m grpc_tools.protoc` -3. **Generated working Python files** (`kas_pb2.py`, `kas_pb2_grpc.py`) -4. **Verified imports and functionality** - all message types are accessible -5. **Created automated build scripts** for easy regeneration -6. **Integrated with main project** via file syncing - -### ✅ Verified Working Features -- ✅ Import: `from otdf_python.proto import kas_pb2, kas_pb2_grpc` -- ✅ Message creation: `req = kas_pb2.RewrapRequest()` -- ✅ gRPC service stubs: `kas_pb2_grpc.AccessServiceStub` -- ✅ All core message types: `RewrapRequest`, `RewrapResponse`, `InfoRequest`, etc. - -### Test Results -```bash -Successfully imported protobuf files -Found 35 symbols in kas_pb2 -Found 19 symbols in kas_pb2_grpc -Available message types: -- RewrapRequest: True -- RewrapResponse: True -- AccessService: True -Created request with token: test -``` - -## Generated Files - -The generated Python files include: -- **`kas_pb2.py`** - Protocol buffer message classes -- **`kas_pb2_grpc.py`** - gRPC service client and server classes - -These files are automatically synced to `otdf-python-proto/generated/` and used by the main project in `src/otdf_python/`. - -## Fallback Strategy - -When the original proto file has missing dependencies (like Google API annotations), the system automatically creates a simplified version that includes all the core message types and services but removes problematic imports. This ensures the build always succeeds and provides usable protobuf classes. diff --git a/otdf-python-proto/README.md b/otdf-python-proto/README.md index 7327a9a..2e4c102 100644 --- a/otdf-python-proto/README.md +++ b/otdf-python-proto/README.md @@ -12,7 +12,7 @@ This project now supports **Connect RPC**, a modern HTTP-friendly alternative to - 🚀 **Simplified deployment** - No special gRPC infrastructure required - 📊 **Better observability** - Standard HTTP status codes and headers -See [CONNECT_RPC_MIGRATION.md](../CONNECT_RPC_MIGRATION.md) for migration guide and examples. +See [CONNECT_RPC.md](../docs/CONNECT_RPC.md) for additional information. ## Structure @@ -158,7 +158,7 @@ response = client.GetPolicy(request) If you're migrating from traditional gRPC clients to Connect RPC: -1. Read the [Connect RPC Migration Guide](../CONNECT_RPC_MIGRATION.md) +1. Read the [Connect RPC Migration Guide](../docs/CONNECT_RPC.md) 2. Run the Connect RPC generation: `./scripts/build_connect_proto.sh` (or from the submodule: `cd otdf-python-proto && uv run python scripts/generate_connect_proto.py`) 3. Update your client code to use `*_connect.py` modules 4. Test with your authentication and deployment setup diff --git a/otdf-python-proto/scripts/build_connect_proto.sh b/otdf-python-proto/scripts/build_connect_proto.sh index d5b4591..3a90859 100755 --- a/otdf-python-proto/scripts/build_connect_proto.sh +++ b/otdf-python-proto/scripts/build_connect_proto.sh @@ -87,7 +87,7 @@ if [[ $? -eq 0 ]]; then echo " python examples/connect_rpc_client_example.py" echo "" echo "For more information, see:" - echo " - CONNECT_RPC_MIGRATION.md" + echo " - docs/CONNECT_RPC.md" echo " - https://connectrpc.com/docs/" else echo "✗ Connect RPC generation failed!"