From 58c7286dc9275e02b8e582344a576c0562756220 Mon Sep 17 00:00:00 2001 From: Mihai Criveti Date: Sun, 21 Sep 2025 12:58:29 +0100 Subject: [PATCH] Documentation updates Signed-off-by: Mihai Criveti --- docs/docs/development/building.md | 7 +- docs/docs/development/developer-onboarding.md | 4 +- .../docs/development/developer-workstation.md | 14 +--- docs/docs/development/doctest-coverage.md | 81 +++++++++---------- docs/docs/development/documentation.md | 4 +- docs/docs/development/github.md | 34 ++++---- docs/docs/development/index.md | 6 +- .../mcp-developer-guide-json-rpc.md | 64 ++++++++------- docs/docs/development/packaging.md | 11 ++- docs/docs/development/review.md | 11 ++- 10 files changed, 115 insertions(+), 121 deletions(-) diff --git a/docs/docs/development/building.md b/docs/docs/development/building.md index 5edb69468..70f018e1f 100644 --- a/docs/docs/development/building.md +++ b/docs/docs/development/building.md @@ -49,8 +49,9 @@ You can run the gateway with: ```bash make serve # production-mode (Gunicorn) on http://localhost:4444 make dev # hot-reload (Uvicorn) on http://localhost:8000 -make run # wrapper over uvicorn; pass --reload to enable auto-reload -./run.sh --reload # equivalent of 'make run' with explicit flags +make run # executes ./run.sh with your current .env settings +RELOAD=true make run # enable auto-reload via run.sh (same as ./run.sh --reload) +./run.sh --help # view all supported flags ``` Use `make dev` during development for auto-reload on port 8000. @@ -59,7 +60,7 @@ Use `make dev` during development for auto-reload on port 8000. ## πŸ”„ Live Reload Tips -Ensure `RELOAD=true` and `DEV_MODE=true` are set in your `.env` during development. +When relying on `run.sh`, set `RELOAD=true` (or pass `--reload`) and `DEV_MODE=true` in your `.env` so settings match. Also set: diff --git a/docs/docs/development/developer-onboarding.md b/docs/docs/development/developer-onboarding.md index a972a943e..8cc0e0fb4 100644 --- a/docs/docs/development/developer-onboarding.md +++ b/docs/docs/development/developer-onboarding.md @@ -17,7 +17,7 @@ ???+ check "Python tooling" - [ ] `pip install --upgrade pip` - [ ] `uv` and `uvx` installed - [install uv](https://github.com/astral-sh/uv) - - [ ] `.venv` created with `make venv install install-dev` + - [ ] `.venv` recreated with `make install-dev` (installs runtime + dev extras) ???+ check "Additional tools" - [ ] `helm` installed for Kubernetes deployments ([Helm install docs](https://helm.sh/docs/intro/install/)) @@ -43,7 +43,7 @@ ???+ check "Local setup" - [ ] `make check-env` (validates .env is complete) - - [ ] `make venv install install-dev serve` + - [ ] `make install-dev serve` - [ ] `make smoketest` runs and passes ???+ check "Container builds" diff --git a/docs/docs/development/developer-workstation.md b/docs/docs/development/developer-workstation.md index 8873a14fa..e5abd54f6 100644 --- a/docs/docs/development/developer-workstation.md +++ b/docs/docs/development/developer-workstation.md @@ -100,16 +100,10 @@ This guide helps you to set up your local environment for contributing to the Mo ### Set Up and Serve Documentation ```bash -# Create and activate virtual environment -make venv -source .venv/bin/activate # Linux/macOS -.venv\Scripts\activate # Windows - -# Install dependencies -make install - -# Serve documentation locally -make serve +# Build docs in an isolated environment +cd docs +make venv # first run only; installs MkDocs + plugins +make serve # http://127.0.0.1:8000 with live reload ``` ## Signing commits diff --git a/docs/docs/development/doctest-coverage.md b/docs/docs/development/doctest-coverage.md index 3cd23a370..3a0be11d6 100644 --- a/docs/docs/development/doctest-coverage.md +++ b/docs/docs/development/doctest-coverage.md @@ -24,28 +24,25 @@ Doctest is a Python testing framework that extracts interactive examples from do ## Coverage Status -### Current Coverage - -| Module Category | Status | Coverage | -|----------------|--------|----------| -| **Transport Modules** | βœ… Complete | 100% | -| **Utility Functions** | βœ… Complete | 100% | -| **Validation Modules** | βœ… Complete | 100% | -| **Configuration** | βœ… Complete | 100% | -| **Service Classes** | πŸ”„ In Progress | ~60% | -| **Complex Classes** | πŸ”„ In Progress | ~40% | - -### Modules with Full Coverage - -- `mcpgateway/transports/base.py` - Base transport interface -- `mcpgateway/transports/stdio_transport.py` - Standard I/O transport -- `mcpgateway/transports/sse_transport.py` - Server-Sent Events transport -- `mcpgateway/transports/websocket_transport.py` - WebSocket transport -- `mcpgateway/transports/streamablehttp_transport.py` - Streamable HTTP transport -- `mcpgateway/transports/__init__.py` - Transport module exports -- `mcpgateway/utils/create_slug.py` - Slug generation utilities -- `mcpgateway/validation/jsonrpc.py` - JSON-RPC validation -- `mcpgateway/config.py` - Configuration management +### Current Focus + +| Area | Status | Notes | +| ---- | ------ | ----- | +| Core transports & utilities | βœ… Doctest examples live directly in the modules (e.g. `mcpgateway/transports/*`, `mcpgateway/config.py`, `mcpgateway/wrapper.py`) | +| Service layer | πŸ”„ Many high-traffic services include doctests, but coverage is being expanded as modules are touched | +| Validators & schemas | βœ… JSON-RPC validation, slug helpers, and schema models ship with doctest-backed examples | +| Remaining modules | 🚧 Add doctests opportunistically when new behaviour is introduced | + +### Key modules with doctests today + +The following modules already contain runnable doctest examples you can reference when adding new ones: + +- `mcpgateway/transports/base.py`, `stdio_transport.py`, `sse_transport.py`, `streamablehttp_transport.py` +- `mcpgateway/cache/session_registry.py` (initialisation handshake and SSE helpers) +- `mcpgateway/config.py` and supporting validators +- `mcpgateway/utils/create_jwt_token.py` +- `mcpgateway/wrapper.py` (URL conversion, logging toggles) +- `mcpgateway/validation/jsonrpc.py` ## Running Doctests @@ -81,9 +78,17 @@ Doctests are automatically run in the GitHub Actions pipeline: ```yaml # .github/workflows/pytest.yml -- name: Run doctests +- name: "πŸ“Š Doctest coverage with threshold" + run: | + pytest --doctest-modules mcpgateway/ \ + --cov=mcpgateway \ + --cov-report=term \ + --cov-report=json:doctest-coverage.json \ + --cov-fail-under=40 \ + --tb=short +- name: "πŸ“Š Doctest coverage validation" run: | - pytest --doctest-modules mcpgateway/ -v + python -m pytest --doctest-modules mcpgateway/ --tb=no -q ``` ## Doctest Standards @@ -157,41 +162,31 @@ def send_message(self, message: Dict[str, Any]) -> None: ## Pre-commit Integration -Doctests are integrated into the pre-commit workflow: +The default `.pre-commit-config.yaml` ships with a doctest hook commented out. Enable it locally by uncommenting the block (or copying the snippet below) if you want doctests to run on every commit: ```yaml -# .pre-commit-config.yaml - repo: local hooks: - id: doctest name: Doctest - entry: pytest --doctest-modules mcpgateway/ + entry: pytest --doctest-modules mcpgateway/ --tb=short language: system types: [python] ``` -This ensures that: -- All doctests pass before commits are allowed -- Documentation examples are always verified -- Code quality is maintained automatically +When enabled, the hook blocks commits until doctests passβ€”handy if you're touching modules with extensive inline examples. ## Coverage Metrics -### Current Statistics - -- **Total Functions/Methods**: ~200 -- **Functions with Doctests**: ~150 -- **Coverage Percentage**: ~75% -- **Test Examples**: ~500+ +- `make doctest-coverage` writes an HTML report to `htmlcov-doctest/` and an XML summary to `coverage-doctest.xml`. +- GitHub Actions currently enforces a doctest coverage floor of **40%** via `--cov-fail-under=40`. +- Use `coverage json -o doctest-coverage.json` (already produced in CI) or `coverage report` locally to inspect specific modules. ### Coverage Goals -- **Phase 1**: βœ… Infrastructure setup (100%) -- **Phase 2**: βœ… Utility modules (100%) -- **Phase 3**: βœ… Configuration and schemas (100%) -- **Phase 4**: βœ… Service classes (100%) -- **Phase 5**: βœ… Transport modules (100%) -- **Phase 6**: πŸ”„ Documentation integration (100%) +1. Keep transports, config validators, and request/response helpers covered with runnable examples. +2. Add doctests alongside new service-layer logic instead of backfilling everything at once. +3. Promote tricky bug fixes into doctest examples so regressions surface quickly. ## Contributing Guidelines diff --git a/docs/docs/development/documentation.md b/docs/docs/development/documentation.md index 60d566ffc..9b1116ae7 100644 --- a/docs/docs/development/documentation.md +++ b/docs/docs/development/documentation.md @@ -145,9 +145,9 @@ The `build` target produces a fully-static site (used by CI for docs previews an ## πŸ“€ Publishing (CI) -Docs are tested, but not deployed automatically by GitHub Actions on every push to `main`. The workflow runs `cd docs && make build`. +We do not currently run a dedicated docs-build workflow in CI. Build locally with `make build` (or the `make doctest`/`make lint` suite from the repo root) before opening a PR that touches docs-heavy changes. -Publishing is done manually by repo maintainers with `make deploy` which publishes the generated site to **GitHub Pages**. +Publishing to GitHub Pages remains a manual maintainer task via `make deploy`. --- diff --git a/docs/docs/development/github.md b/docs/docs/development/github.md index 7ff3106a1..242ff6446 100644 --- a/docs/docs/development/github.md +++ b/docs/docs/development/github.md @@ -192,7 +192,7 @@ gh pr checkout 29 ### 5.1 Local build (SQLite + self-signed HTTPS) ```bash -make venv install install-dev serve-ssl +make install-dev serve-ssl ``` * Sets up a Python virtualenv @@ -202,12 +202,13 @@ make venv install install-dev serve-ssl ### 5.2 Container build (PostgreSQL + Redis) ```bash -make compose-up +make docker-prod # build the lite runtime image locally +make compose-up # start the Docker Compose stack (PostgreSQL + Redis) ``` * Spins up the full Docker Compose stack * Uses PostgreSQL for persistence and Redis for queueing -* Rebuilds images so you catch Docker-specific issues +* Rebuilds/uses your freshly built image so you catch Docker-specific issues ### 5.3 Gateway JWT (local API access) @@ -215,21 +216,21 @@ Quickly confirm that authentication works and the gateway is healthy: ```bash export MCPGATEWAY_BEARER_TOKEN=$(python3 -m mcpgateway.utils.create_jwt_token -u admin@example.com --secret my-test-key) -curl -s -k -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" https://localhost:4444/health +curl -s -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" http://localhost:4444/health ``` Expected output: ```json -{"status": "ok"} +{"status": "healthy"} ``` -If you see anything other than `{"status":"ok"}`, investigate before approving the PR. +If you see anything other than `{"status":"healthy"}`, investigate before approving the PR. Quickly confirm that the MCP Gateway is configured with the correct database, and it is reachable: ```bash -curl -s -k -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" https://localhost:4444/version | jq +curl -s -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" http://localhost:4444/version | jq ``` Then proceed to register an MCP Server under Gateways using the UI, ensuring that Tools work, creating a Virtual Server and testing that from UI, API and a MCP Client. @@ -311,19 +312,14 @@ Use the UI method only if reviewers are done-every push re-triggers CI. Before requesting review, confirm that **all** required status checks on the PR page are green βœ… ("All checks have passed"). You should now see something like: ```text -Bandit / bandit (pull_request) βœ… Successful in 21s -Build Python Package / build-package (3.10) βœ… Successful in 12s -Code scanning results / Bandit βœ… No new alerts in code changed by this pull request -Code scanning results / Dockle βœ… No new alerts in code changed by this pull request -Code scanning results / Hadolint βœ… No new alerts in code changed by this pull request -Code scanning results / Trivy βœ… No new alerts in code changed by this pull request -CodeQL Advanced / CodeQL (javascript-typescript)βœ… Successful in 1m -CodeQL Advanced / CodeQL (python) βœ… Successful in 1m +Secure Docker Build / docker-image βœ… Successful in ~4m +Build Python Package / python-package (3.11) βœ… Successful in ~2m +Tests & Coverage / pytest βœ… Successful in ~3m +Lint & Static Analysis / lint βœ… Successful in ~2m +Bandit / bandit βœ… Successful in ~1m +CodeQL βœ… Successful in ~1m +Dependency Review / dependency-review βœ… Successful in seconds DCO βœ… Passed -Dependency Review / dependency-review βœ… Successful in 4s -Secure Docker Build / build-scan-sign βœ… Successful in 4m -Travis CI - Branch βœ… Build Passed -Travis CI - Pull Request βœ… Build Passed ``` If anything is red or still running, wait or push a **fix in the same PR** until every line is green. Ensure that a CODEOWNER is assigned to review the PR. diff --git a/docs/docs/development/index.md b/docs/docs/development/index.md index 86cec2f8d..7892a0ec4 100644 --- a/docs/docs/development/index.md +++ b/docs/docs/development/index.md @@ -103,9 +103,9 @@ echo $MCPGATEWAY_BEARER_TOKEN Then test: ```bash -curl -k -sX GET \ +curl -sX GET \ -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ - https://localhost:4444/tools | jq + http://localhost:4444/tools | jq ``` --- @@ -126,7 +126,7 @@ Key configs include: | ------------------- | ---------------------------- | | `DATABASE_URL` | Database connection | | `JWT_SECRET_KEY` | Signing key for JWTs | -| `DEV_MODE=true` | Enables hot reload and debug | +| `DEV_MODE=true` | Enables relaxed development defaults (set together with `RELOAD=true` if you rely on `run.sh`) | | `CACHE_TYPE=memory` | Options: memory, redis, none | --- diff --git a/docs/docs/development/mcp-developer-guide-json-rpc.md b/docs/docs/development/mcp-developer-guide-json-rpc.md index 55f599b7f..d5b97aa98 100644 --- a/docs/docs/development/mcp-developer-guide-json-rpc.md +++ b/docs/docs/development/mcp-developer-guide-json-rpc.md @@ -35,9 +35,7 @@ curl -s -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ **Expected health response:** ```json { - "status": "healthy", - "timestamp": "2025-01-15T10:30:00Z", - "version": "0.7.0" + "status": "healthy" } ``` @@ -118,22 +116,16 @@ curl -X POST -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ "result": { "protocolVersion": "2025-03-26", "capabilities": { - "experimental": {}, - "prompts": { - "listChanged": false - }, - "resources": { - "subscribe": false, - "listChanged": false - }, - "tools": { - "listChanged": false - } + "prompts": {"listChanged": true}, + "resources": {"subscribe": true, "listChanged": true}, + "tools": {"listChanged": true}, + "logging": {} }, "serverInfo": { - "name": "mcpgateway", + "name": "MCP_Gateway", "version": "0.7.0" - } + }, + "instructions": "MCP Gateway providing federated tools, resources and prompts. Use /admin interface for configuration." } } ``` @@ -182,7 +174,7 @@ curl -X POST -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ } ``` -**Response with tools available:** +**Response with tools available (e.g. after registering the Fast Time Server example):** ```json { "jsonrpc": "2.0", @@ -396,9 +388,22 @@ curl -N -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ http://localhost:4444/sse ``` +The first event emitted by the stream is an `endpoint` payload with the per-session POST URL: + +```text +event: endpoint +data: http://localhost:4444/message?session_id=7bfbf2a4-... +``` + +Copy that value (it changes each run) into an environment variable used by your second terminal: + +```bash +export MCP_SSE_ENDPOINT="http://localhost:4444/message?session_id=7bfbf2a4-..." +``` + ### Sending Messages via SSE -In a separate terminal, send JSON-RPC messages: +Now send JSON-RPC messages to the captured endpoint: ```bash # Initialize via SSE @@ -414,7 +419,7 @@ curl -X POST -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ "clientInfo": {"name": "sse-client", "version": "1.0"} } }' \ - http://localhost:4444/message + "$MCP_SSE_ENDPOINT" # List tools via SSE curl -X POST -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ @@ -424,9 +429,9 @@ curl -X POST -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ "id": 2, "method": "tools/list" }' \ - http://localhost:4444/message + "$MCP_SSE_ENDPOINT" -# Call a tool via SSE +# Call a tool via SSE (after registering one) curl -X POST -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ -H "Content-Type: application/json" \ -d '{ @@ -438,7 +443,7 @@ curl -X POST -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \ "arguments": {"timezone": "Asia/Tokyo"} } }' \ - http://localhost:4444/message + "$MCP_SSE_ENDPOINT" ``` ## Utility Operations @@ -485,17 +490,18 @@ python3 -m mcpgateway.wrapper ### STDIO Communication -Send JSON-RPC commands directly to stdin: +Feed multiple JSON-RPC messages in one stream (the wrapper exits when STDIN closes): ```bash -# Send commands to stdin (each on a single line) -echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"stdio-client","version":"1.0"}}}' | python3 -m mcpgateway.wrapper - -echo '{"jsonrpc":"2.0","method":"notifications/initialized","params":{}}' | python3 -m mcpgateway.wrapper - -echo '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | python3 -m mcpgateway.wrapper +python3 -m mcpgateway.wrapper <<'EOF' +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"stdio-client","version":"1.0"}}} +{"jsonrpc":"2.0","method":"notifications/initialized","params":{}} +{"jsonrpc":"2.0","id":2,"method":"tools/list"} +EOF ``` +Run it interactively (without the here-doc) if you prefer to type requests by hand. + ## Complete Session Examples ### HTTP JSON-RPC Complete Session diff --git a/docs/docs/development/packaging.md b/docs/docs/development/packaging.md index f8f618648..25e2511cf 100644 --- a/docs/docs/development/packaging.md +++ b/docs/docs/development/packaging.md @@ -9,14 +9,15 @@ This guide covers how to package MCP Gateway for deployment in various environme Build an OCI-compliant container image using: ```bash -make podman +make podman # builds using Containerfile with Podman +# or manually podman build -t mcpgateway:latest -f Containerfile . ``` Or with Docker (if Podman is not available): ```bash -make docker +make docker # builds using Containerfile with Docker # or manually docker build -t mcpgateway:latest -f Containerfile . ``` @@ -59,13 +60,15 @@ You can bump the version manually or automate it via Git tags or CI/CD. ## πŸ“ Release Artifacts -If you need to ship ZIPs, wheels, or a full binary: +If you need to ship ZIPs or wheels use the project build tooling: ```bash +make dist +# or python3 -m build ``` -Outputs will be under `dist/`. You can then: +Outputs land under `dist/`. You can then: * Push to PyPI (internal or public) * Upload to GitHub Releases diff --git a/docs/docs/development/review.md b/docs/docs/development/review.md index d28147616..023c0fd03 100644 --- a/docs/docs/development/review.md +++ b/docs/docs/development/review.md @@ -46,20 +46,19 @@ Before you read code or leave comments, **always** verify the PR builds and test ### 3.1 Local Build ```bash -make venv install install-dev serve # Install into a fresh venv, and test it runs locally +make install-dev serve # Install into a fresh venv, and test it runs locally ``` ### 3.2 Container Build and testing with Postgres and Redis (compose) ```bash -make docker-prod # Build a new image -# Change: image: mcpgateway/mcpgateway:latest in docker-compose.yml to use the local image -make compose-up # spins up the Docker Compose stack +make docker-prod # Build the lite runtime image locally +make compose-up # Spins up the Docker Compose stack (PostgreSQL + Redis) # Test the basics -curl -k https://localhost:4444/health` # {"status":"healthy"} +curl http://localhost:4444/health # {"status":"healthy"} export MCPGATEWAY_BEARER_TOKEN=$(python3 -m mcpgateway.utils.create_jwt_token --username admin@example.com --exp 0 --secret my-test-key) -curl -sk -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" http://localhost:4444/version | jq -c '.database, .redis' +curl -s -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" http://localhost:4444/version | jq -c '.database, .redis' # Add an MCP server to http://localhost:4444 then check logs: make compose-logs