gRPC Internal Transport

gRPC Internal Communication & Transport

OpenCrawling supports gRPC as an optional, high-performance binary transport protocol for internal component and payload communication across microservices (e.g. oc-repository-connectors, oc-core, oc-worker, oc-output-connectors). By default, the transport mode is configured to AUTO.

To guarantee backward compatibility, seamless local development, and zero-downtime upgrades, gRPC operates alongside automatic HTTP/REST fallback capabilities. Administrators can dynamically toggle transport modes, set payload limits, configure TLS/mTLS encryption, and execute live diagnostic connection tests directly from the Internal Communication & Transport Settings Panel in the OpenCrawling Admin Dashboard.

---

🎯 Motivation & Performance Benefits

While standard HTTP/1.1 REST/JSON is ideal for simple single-node deployments and local development, high-throughput enterprise ingestion workloads require lower network overhead and higher payload efficiency:

1. Up to 50% Payload Size Reduction: Protobuf binary serialization significantly compresses raw document metadata and chunk payload transport compared to verbose JSON strings.
2. Reduced CPU & Memory Overhead: HTTP/2 multiplexing and streaming stubs avoid expensive HTTP/1.1 connection pooling overhead and JSON parsing/serialization cycles.
3. Operational Resilience (AUTO Mode): Automatic fallback to HTTP/REST ensures ingestion jobs continue uninterrupted even if network firewalls, service meshes, or intermediate nodes temporarily block gRPC channels.
4. Zero-Downtime Upgrades: Existing OpenCrawling clusters can be upgraded node-by-node without requiring simultaneous infrastructure changes across all nodes.
5. Centralized Admin Management: Dynamic protocol toggling, port re-assignment, and live diagnostic ping tests are exposed via Admin UI and REST APIs without requiring server restarts.

---

🏗️ Architecture & Component Flow

graph TD
    subgraph Admin Dashboard
        UI[Admin Panel - Transport Settings UI]
    end

    subgraph oc-runtime / Management API
        API[Transport Controller - /api/v1/admin/settings/transport]
        TM[Internal Transport Manager]
    end

    subgraph Internal Transport Channels
        gRPC[gRPC Server/Client Channel - Port 9095]
        REST[HTTP/REST Fallback Channel]
    end

    subgraph Microservice Pipeline
        RC[oc-repository-connector]
        WORKER[oc-worker / Processing Node]
        OUT[oc-output-connector]
    end

    UI -->|1. Configure Mode: AUTO / GRPC / REST| API
    API -->|2. Dynamic Hot-Reload Settings| TM
    
    RC -->|3. Send Document Payload| TM
    TM -->|Mode = AUTO or GRPC| gRPC
    gRPC -->|Channel Error / Timeout| REST
    TM -->|Mode = REST| REST
    
    gRPC --> WORKER
    REST --> WORKER
    WORKER --> OUT

Loading

Protocol Modes

Mode	Behaviour	Best Used For
AUTO (Default / Recommended)	Attempts high-speed gRPC binary streaming first. If the gRPC channel fails or times out, it automatically falls back to HTTP/REST transport.	Production environments, hybrid cloud deployments, zero-downtime upgrades
GRPC (Strict)	Enforces binary Protobuf gRPC streaming across all processing nodes. Payload delivery fails if gRPC is unreachable.	High-throughput enterprise clusters with verified gRPC networking
REST (Standard)	Disables internal gRPC server and uses standard HTTP/REST JSON endpoints.	Simple single-node deployments, local testing, environments restricting gRPC

---

📜 Protobuf Schema Contract (oc-grpc-api)

The internal payload communication contract is defined in the oc-grpc-api module under src/main/proto/opencrawling/internal/v1/internal_payload.proto:

syntax = "proto3";

package opencrawling.internal.v1;

option java_multiple_files = true;
option java_package = "org.opencrawling.internal.v1";
option java_outer_classname = "InternalPayloadProto";

service InternalPayloadService {
  // Bi-directional document streaming
  rpc StreamDocumentPayloads (stream DocumentPayloadRequest) returns (stream PayloadIngestionResponse);

  // Single document payload ingestion
  rpc SendDocumentPayload (DocumentPayloadRequest) returns (PayloadIngestionResponse);

  // Health check / probe for Admin UI diagnostic tests
  rpc PingTransport (PingRequest) returns (PingResponse);
}

message DocumentPayloadRequest {
  string task_id = 1;
  string repository_id = 2;
  string document_id = 3;
  bytes raw_content = 4;
  map<string, string> metadata = 5;
  repeated string security_acls = 6;
  int64 timestamp = 7;
}

message PayloadIngestionResponse {
  string document_id = 1;
  enum Status { SUCCESS = 0; PROCESSING = 1; FAILED = 2; }
  Status status = 2;
  string error_message = 3;
}

message PingRequest { int64 client_timestamp = 1; }
message PingResponse { int64 server_timestamp = 1; bool healthy = 2; }

---

⚙️ Configuration Parameters

Internal transport settings can be configured via Spring Boot application.yml, environment variables, or dynamically via the Admin UI / REST API.

1. Spring Boot Properties (application.yml)

opencrawling:
  transport:
    mode: AUTO                         # Options: AUTO, GRPC, REST
    grpc:
      enabled: true                    # Enable/disable gRPC server lifecycle
      port: 9095                       # gRPC server port (default: 9095)
      max-message-size-mb: 32          # Maximum inbound message size in MB (default: 32 MB)
      fallback-to-rest: true           # Enable automatic fallback to REST on gRPC failure
      connection-timeout-ms: 5000      # gRPC connection deadline timeout in ms
      keep-alive-time-ms: 30000        # Keep-alive ping interval in ms
      tls:
        enabled: false                 # Enable TLS / mTLS transport encryption
        cert-chain-path: ""            # Path to server TLS certificate chain (.crt / .pem)
        private-key-path: ""           # Path to private key file (.key)

2. Environment Variables

Environment Variable	Property Key	Default Value	Description
OPENCRAWLING_TRANSPORT_MODE	opencrawling.transport.mode	AUTO	Active transport mode (AUTO, GRPC, REST)
OPENCRAWLING_TRANSPORT_GRPC_PORT	opencrawling.transport.grpc.port	9095	Internal gRPC listener port
OPENCRAWLING_TRANSPORT_GRPC_ENABLED	opencrawling.transport.grpc.enabled	true	Enables or disables gRPC transport server

---

🔒 Security & Transport Encryption (TLS / mTLS)

For secured enterprise environments, internal gRPC channels can be encrypted using TLS / mTLS:

1. Enable TLS in application.yml or via the Admin UI Settings panel (tlsEnabled=true).
2. Provide absolute paths to the server certificate chain and private key:
   • certChainPath: /etc/ssl/certs/opencrawling-grpc.crt
   • privateKeyPath: /etc/ssl/certs/opencrawling-grpc.key
3. InternalTransportManager will initialize the Netty-backed gRPC server with transport security enabled. If certificate files are missing or unreadable, a warning is logged and execution safely falls back to plaintext or REST.

---

🖥️ Management REST API Endpoints

The Admin UI and external management tools interact with gRPC transport configuration via TransportSettingsController:

1. Fetch Transport Settings & Health

• Endpoint: GET /api/v1/admin/settings/transport (or /api/system/settings/transport)
• Response:

{
  "status": "UP",
  "activeMode": "AUTO",
  "grpcEnabled": true,
  "grpcPort": 9095,
  "fallbackToRest": true,
  "settings": {
    "mode": "AUTO",
    "grpcEnabled": true,
    "grpcPort": 9095,
    "maxMessageSizeMb": 32,
    "fallbackToRest": true,
    "keepAliveTimeMs": 30000,
    "connectionTimeoutMs": 5000,
    "tlsEnabled": false,
    "certChainPath": "",
    "privateKeyPath": ""
  }
}

2. Update Transport Configuration (Hot Reload)

• Endpoint: PUT /api/v1/admin/settings/transport
• Body: Updated TransportSettingsDTO JSON payload.
• Action: Dynamically reconfigures and restarts the underlying gRPC server channel without restarting oc-runtime.

3. Live Diagnostic Connection Test

• Endpoint: POST /api/v1/admin/settings/transport/test-grpc
• Body: {"host": "127.0.0.1", "port": 9095}
• Action: Triggers a live gRPC PingTransport RPC probe against target host and port to measure channel latency and verify handshake health before saving settings.
• Response:

{
  "status": "SUCCESS",
  "grpcAvailable": true,
  "restFallbackActive": true,
  "latencyMs": 4,
  "message": "gRPC connection test successful! Server online on 127.0.0.1:9095 (Latency: 4 ms)"
}

---

🖥️ Admin UI Controls

Administrators can configure and test gRPC transport from Section 4: Internal Communication & Transport (gRPC / REST) in the Admin UI Settings view:

• Mode Selector Cards: Interactively switch between AUTO, GRPC (Strict), and REST (Standard).
• Config Fields: Adjust gRPC Server Port, Max Message Size (MB), and Connection Timeout (ms).
• Resilience Options: Toggle Auto-fallback to HTTP/REST on Channel Timeout or Error and Enable TLS / mTLS Transport Encryption.
• Live Test Button: Click "Test gRPC Connectivity" to execute a diagnostic probe and view real-time latency badges (SUCCESS, FALLBACK, or FAILED).

---

🧪 Integration Testing & Verification

The gRPC transport layer can be verified using Maven test commands:

# Run isolated gRPC transport manager and protocol unit tests
mvn test -pl oc-grpc-api,oc-core,oc-runtime -Dtest=InternalTransportManagerTest

To test diagnostic connectivity via curl:

# Diagnostic probe to test gRPC ping on localhost:9095
curl -X POST "http://localhost:8080/api/v1/admin/settings/transport/test-grpc" \
  -H "Content-Type: application/json" \
  -d '{"host":"127.0.0.1","port":9095}'

Docker Container Builds

All multi-stage Java container Dockerfiles (oc-runtime/Dockerfile, docker/Dockerfile.crawler, docker/Dockerfile.ingestion-consumer, docker/Dockerfile.embedding-consumer, docker/Dockerfile.writer-consumer, docker/Dockerfile.mcp-server) include oc-grpc-api/pom.xml and oc-grpc-api/src/ in the build stage to compile the Protobuf stubs during image generation.