Feature/oauth handler

docs/integrations/app-integrations/rest-api.mdx

@@ -0,0 +1,148 @@
+---
+title: REST API
+sidebarTitle: REST API
+---
+
+In this section, we present how to connect any REST API to MindsDB using bearer-token authentication.
+
+The REST API handler is a generic integration that lets you forward HTTP requests to any API through MindsDB using stored credentials. Unlike named integrations (HubSpot, Shopify, etc.), it requires no handler-specific knowledge — just a base URL and a bearer token.
+
+This is useful for APIs that MindsDB doesn't have a dedicated handler for, or when you only need direct HTTP access without SQL table mapping.
+
+## Connection
+
+The required arguments to establish a connection are as follows:
+
+- `base_url`: the base URL of the REST API (e.g. `https://api.example.com`). All request paths are appended to this URL.
+- `bearer_token`: the token used for authentication. Injected as `Authorization: Bearer <token>` on every request.
+
+Optional arguments:
+
+- `default_headers`: a JSON object of static headers added to every request (e.g. `{"Accept": "application/json"}`).
+- `allowed_hosts`: a list of allowed hostnames for requests. Defaults to the hostname of `base_url`. Use `["*"]` to disable host containment.
+- `test_path`: the path used by the test endpoint to verify connectivity. Defaults to `/`.
+
+To connect a REST API to MindsDB, create a new database:
+
+```sql
+CREATE DATABASE my_api
+WITH ENGINE = 'rest_api',
+PARAMETERS = {
+  "base_url": "https://api.example.com",
+  "bearer_token": "your_token_here"
+};
+```
+
+### Example: Connect to HubSpot
+
+```sql
+CREATE DATABASE my_hubspot
+WITH ENGINE = 'rest_api',
+PARAMETERS = {
+  "base_url": "https://api.hubapi.com",
+  "bearer_token": "pat-eu1-..."
+};
+```
+
+### Example: Connect with default headers and a custom test path
+
+```sql
+CREATE DATABASE my_internal_api
+WITH ENGINE = 'rest_api',
+PARAMETERS = {
+  "base_url": "https://internal.example.com/api/v2",
+  "bearer_token": "sk-...",
+  "default_headers": {"Accept": "application/json"},
+  "test_path": "/health"
+};
+```
+
+### Example: Multiple allowed hosts
+
+```sql
+CREATE DATABASE my_multi_region_api
+WITH ENGINE = 'rest_api',
+PARAMETERS = {
+  "base_url": "https://api.example.com",
+  "bearer_token": "your_token",
+  "allowed_hosts": ["api.example.com", "api.eu.example.com"]
+};
+```
+
+## Usage
+
+This handler is **passthrough-only** — it does not expose SQL tables. All interaction is through the REST passthrough endpoint.
+
+### Sending requests
+
+Forward HTTP requests to the upstream API:
+
+```
+POST /api/integrations/my_api/passthrough
+```
+
+```json
+{
+  "method": "GET",
+  "path": "/v1/users",
+  "query": {"limit": "10"},
+  "headers": {"Accept": "application/json"}
+}
+```
+
+The response wraps the upstream HTTP response:
+
+```json
+{
+  "status_code": 200,
+  "headers": {"content-type": "application/json"},
+  "body": {"results": [...]},
+  "content_type": "application/json"
+}
+```
+
+Supported HTTP methods: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`.
+
+### Testing the connection
+
+Verify that the base URL, token, and host allowlist are configured correctly:
+
+```
+POST /api/integrations/my_api/passthrough/test
+```
+
+A successful response:
+
+```json
+{"ok": true, "status_code": 200, "host": "api.example.com", "latency_ms": 140}
+```
+
+A failed response:
+
+```json
+{"ok": false, "error_code": "auth_failed", "message": "upstream rejected credentials; base URL and allowlist look correct"}
+```
+
+## Security
+
+- Credentials are stored in MindsDB and never exposed to the caller.
+- Requests are restricted to hostnames in the allowlist. Private and loopback IP addresses are rejected by default.
+- Callers cannot override `Authorization`, `Host`, `Cookie`, or `Proxy-*` headers.
+- If the upstream API echoes the token in responses, it is replaced with `[REDACTED_API_KEY]`.
+- Request bodies are capped at 1 MB, response bodies at 10 MB.
+
+<Warning>
+**`host 'X' is not in the datasource allowlist`**
+
+The request path resolved to a different hostname than `base_url`. Add the hostname to `allowed_hosts`, or use `["*"]` to disable host containment (not recommended for production).
+</Warning>
+
+<Warning>
+**`upstream rejected credentials (401/403)`**
+
+The token is invalid, expired, or missing required scopes. Verify the token with the upstream API provider.
+</Warning>
+
+<Info>
+For more information about available actions and development plans, visit [this page](https://github.com/mindsdb/mindsdb/blob/main/mindsdb/integrations/handlers/rest_api_handler/README.md).
+</Info>

mindsdb/api/http/initialize.py

@@ -28,6 +28,7 @@
 from mindsdb.api.http.namespaces.default import ns_conf as default_ns, check_session_auth
 from mindsdb.api.http.namespaces.file import ns_conf as file_ns
 from mindsdb.api.http.namespaces.handlers import ns_conf as handlers_ns
+from mindsdb.api.http.namespaces.integrations import ns_conf as integrations_ns
 from mindsdb.api.http.namespaces.knowledge_bases import ns_conf as knowledge_bases_ns
 from mindsdb.api.http.namespaces.models import ns_conf as models_ns
 from mindsdb.api.http.namespaces.projects import ns_conf as projects_ns
@@ -280,6 +281,7 @@ def root_index(path):
         agents_ns,
         jobs_ns,
         knowledge_bases_ns,
+        integrations_ns,
     ]
 
     for ns in protected_namespaces:

mindsdb/api/http/namespaces/configs/integrations.py

@@ -0,0 +1,3 @@
+from flask_restx import Namespace
+
+ns_conf = Namespace("integrations", description="API for integration-level operations (passthrough, capabilities)")

mindsdb/api/http/namespaces/integrations.py

@@ -0,0 +1,197 @@
+from http import HTTPStatus
+
+from flask import request
+from flask_restx import Resource
+
+from mindsdb.api.http.utils import http_error
+from mindsdb.api.http.namespaces.configs.integrations import ns_conf
+from mindsdb.api.mysql.mysql_proxy.classes.fake_mysql_proxy import FakeMysqlProxy
+from mindsdb.integrations.libs.passthrough import PassthroughProtocol
+from mindsdb.integrations.libs.passthrough_types import (
+    ALLOWED_METHODS,
+    FORBIDDEN_REQUEST_HEADERS,
+    PassthroughError,
+    PassthroughNotSupportedError,
+    PassthroughRequest,
+    PassthroughResponse,
+    PassthroughValidationError,
+)
+from mindsdb.interfaces.database.integrations import integration_controller
+from mindsdb.metrics.metrics import api_endpoint_metrics
+from mindsdb.utilities import log
+
+logger = log.getLogger(__name__)
+
+
+def _handler_supports_passthrough(handler_module) -> bool:
+    handler_cls = getattr(handler_module, "Handler", None)
+    if handler_cls is None:
+        return False
+    # issubclass is the right check for Protocol when classes define the
+    # methods as real methods (not just dynamic attrs); runtime_checkable
+    # Protocols support issubclass in that mode.
+    try:
+        return issubclass(handler_cls, PassthroughProtocol)
+    except TypeError:
+        return False
+
+
+def _resolve_auth_modes(handler_cls) -> list[str]:
+    """Resolve a handler's advertised auth modes for /capabilities.
+
+    Order of preference:
+        1. `_auth_modes` (list[str]) if defined and non-empty — handlers
+           that support more than one mode (e.g. rest_api: bearer + OAuth
+           client credentials).
+        2. `_auth_mode` (str) if defined — single-mode handlers; preserves
+           the legacy declaration.
+        3. ["bearer"] default — protocol-only handlers that don't declare
+           anything still land in the right bucket.
+    """
+    if handler_cls is None:
+        return ["bearer"]
+    modes = getattr(handler_cls, "_auth_modes", None)
+    if isinstance(modes, (list, tuple)) and modes:
+        return [str(m) for m in modes]
+    mode = getattr(handler_cls, "_auth_mode", None)
+    if isinstance(mode, str) and mode:
+        return [mode]
+    return ["bearer"]
+
+
+def _get_passthrough_handler(name: str):
+    """Look up the datasource's handler and verify it satisfies the contract."""
+    proxy = FakeMysqlProxy()
+    handler = proxy.session.integration_controller.get_data_handler(name)
+    if not isinstance(handler, PassthroughProtocol):
+        raise PassthroughNotSupportedError(f"datasource '{name}' does not support REST passthrough")
+    return handler
+
+
+def _parse_passthrough_request(payload: dict) -> PassthroughRequest:
+    if not isinstance(payload, dict):
+        raise PassthroughValidationError("request body must be a JSON object")
+
+    method = payload.get("method")
+    path = payload.get("path")
+    if not isinstance(method, str) or method.upper() not in ALLOWED_METHODS:
+        raise PassthroughValidationError(f"'method' must be one of {sorted(ALLOWED_METHODS)}")
+    if not isinstance(path, str) or not path.startswith("/"):
+        raise PassthroughValidationError("'path' must be a string starting with '/'")
+
+    headers = payload.get("headers") or {}
+    if not isinstance(headers, dict):
+        raise PassthroughValidationError("'headers' must be an object")
+    for name in headers:
+        if not isinstance(name, str):
+            raise PassthroughValidationError("header names must be strings")
+        if name.lower() in FORBIDDEN_REQUEST_HEADERS or name.lower().startswith("proxy-"):
+            raise PassthroughValidationError(f"header '{name}' is not allowed in passthrough requests")
+
+    query = payload.get("query") or {}
+    if not isinstance(query, dict):
+        raise PassthroughValidationError("'query' must be an object")
+
+    return PassthroughRequest(
+        method=method.upper(),
+        path=path,
+        query={str(k): str(v) for k, v in query.items()},
+        headers={str(k): str(v) for k, v in headers.items()},
+        body=payload.get("body"),
+    )
+
+
+def _serialize_response(resp: PassthroughResponse) -> dict:
+    return {
+        "status_code": resp.status_code,
+        "headers": resp.headers,
+        "body": resp.body,
+        "content_type": resp.content_type,
+    }
+
+
+def _passthrough_error_response(err: PassthroughError):
+    return {
+        "error_code": err.error_code,
+        "message": str(err),
+    }, err.http_status
+
+
+@ns_conf.route("/<name>/passthrough")
+@ns_conf.param("name", "Datasource name")
+class Passthrough(Resource):
+    @ns_conf.doc("passthrough")
+    @api_endpoint_metrics("POST", "/integrations/passthrough")
+    def post(self, name: str):
+        payload = request.json or {}
+        try:
+            req = _parse_passthrough_request(payload)
+            handler = _get_passthrough_handler(name)
+            response = handler.api_passthrough(req)
+        except PassthroughError as e:
+            return _passthrough_error_response(e)
+        except Exception as e:  # noqa: BLE001
+            logger.exception("passthrough failed for datasource %s", name)
+            return http_error(HTTPStatus.INTERNAL_SERVER_ERROR, "PassthroughError", str(e))
+
+        return _serialize_response(response), 200
+
+
+@ns_conf.route("/<name>/passthrough/test")
+@ns_conf.param("name", "Datasource name")
+class PassthroughTest(Resource):
+    @ns_conf.doc("passthrough_test")
+    @api_endpoint_metrics("POST", "/integrations/passthrough/test")
+    def post(self, name: str):
+        try:
+            handler = _get_passthrough_handler(name)
+        except PassthroughError as e:
+            return _passthrough_error_response(e)
+        except Exception as e:  # noqa: BLE001
+            logger.exception("passthrough test lookup failed for datasource %s", name)
+            return http_error(HTTPStatus.INTERNAL_SERVER_ERROR, "PassthroughError", str(e))
+
+        result = handler.test_passthrough()
+        return result, 200
+
+
+@ns_conf.route("/capabilities")
+class Capabilities(Resource):
+    """Return structured passthrough capabilities per handler.
+
+    The new ``handlers`` dict is the canonical shape callers should migrate
+    to. The legacy flat ``bearer_passthrough`` list is still populated for
+    backward compat — Minds can migrate on its own timeline.
+    """
+
+    @ns_conf.doc("integration_capabilities")
+    @api_endpoint_metrics("GET", "/integrations/capabilities")
+    def get(self):
+        handlers: dict[str, dict] = {}
+        bearer_engines: list[str] = []
+        handler_modules = getattr(integration_controller, "handler_modules", {}) or {}
+        for engine, module in handler_modules.items():
+            try:
+                if not _handler_supports_passthrough(module):
+                    continue
+                handler_cls = getattr(module, "Handler", None)
+                # Resolve the handler's advertised auth modes — supports
+                # both the new list-shaped `_auth_modes` and the legacy
+                # single-mode `_auth_mode` declaration.
+                auth_modes = _resolve_auth_modes(handler_cls)
+                handlers[engine] = {
+                    "auth_modes": auth_modes,
+                    "operations": ["passthrough"],
+                }
+                if "bearer" in auth_modes:
+                    bearer_engines.append(engine)
+            except Exception:
+                # A broken handler module should not break the capabilities endpoint.
+                logger.debug("skipping handler %s during capability probe", engine, exc_info=True)
+        bearer_engines.sort()
+        return {
+            "handlers": handlers,
+            # TODO: remove in v2 once Minds has migrated to the `handlers`
+            # structured shape. Keep backward-compat for now.
+            "bearer_passthrough": bearer_engines,
+        }, 200