Skip to content

Auto Narrativization Copilot

Jump to bottom
Piergiorgio Lucidi edited this page Jul 23, 2026 · 1 revision

Auto-Narrativization Copilot

OpenCrawling's Auto-Narrativization Copilot is an AI-powered feature that automatically generates human-readable Mustache narrative templates and mock datasets from structured connector schemas. It bridges the gap between raw machine data (field names, types, descriptions) and meaningful natural language representations that can be applied to any document transformation pipeline.

Overview

The feature is exposed as a single REST endpoint in oc-runtime:

POST /api/transformation/copilot/generate

Given a schema context (connector type + list of fields), it returns a Mustache template string and a typed mock dataset — which can be immediately fed into the MustacheTransformationConnector to render narrative text from live RepositoryDocument metadata.

Pipeline Flow

Schema Context
  connectorType: "iceberg"
  fields: [id (STRING), amount (DOUBLE), region (STRING), timestamp (DATE)]
           │
           ▼
  TemplateGenerationCopilot (oc-runtime)
    ├── Primary:  Calls Spring AI → Ollama (default) or OpenAI
    └── Fallback: Deterministic template generator (offline-safe)
           │
           ▼
  TemplateCopilotResponse
    ├── template  →  "Record {{id}} processed {{amount}} in {{region}} on {{timestamp}}."
    └── mockData  →  { "id": "Sample id", "amount": 123.45, "region": "Sample region", "timestamp": "2026-01-01" }
           │
           ▼
  MustacheTransformationConnector (oc-core)
    └── Renders template against RepositoryDocument.metadata()
           │
           ▼
  Document content stream = enriched narrative text

Components

Component Module Description
MustacheTransformationConnector oc-core Accepts a Mustache template string, renders it against the metadata map of any RepositoryDocument using JMustache 1.16, and replaces the document's content stream with the rendered text
ConnectorSchema oc-core Immutable Java record holding a connector's schema: connectorType (String) and a list of SchemaField(name, type, description)
RepositoryConnector.getSchema() oc-core New default SPI method; returns ConnectorSchema.UNKNOWN unless overridden
IcebergRepositoryConnector.getSchema() oc-iceberg-repository-connector Overrides getSchema() to introspect the live Apache Iceberg table schema and return a ConnectorSchema with actual field definitions
TemplateGenerationCopilot oc-runtime Spring component that wraps Spring AI ChatModel calls; reads spring.ai.copilot.engine (default: ollama) to select the active model, falls back to deterministic generation when the model is unavailable
NarrativizationCopilotController oc-runtime @RestController exposing POST /api/transformation/copilot/generate
SchemaContextRequest oc-runtime Request DTO: { connectorType: String, fields: List<FieldDto(name, type, description)> }
TemplateCopilotResponse oc-runtime Response DTO record: { template: String, mockData: Map<String, Object> }

REST API Reference

POST /api/transformation/copilot/generate

Generates a Mustache narrative template and mock data for the given schema.

Request Body:

{
  "connectorType": "iceberg",
  "fields": [
    { "name": "id",        "type": "STRING", "description": "Primary record identifier" },
    { "name": "amount",    "type": "DOUBLE", "description": "Transaction monetary value" },
    { "name": "region",    "type": "STRING", "description": "Geographical region code" },
    { "name": "timestamp", "type": "DATE",   "description": "Transaction timestamp" }
  ]
}

Response 200 OK:

{
  "template": "Record {{id}} processed {{amount}} in {{region}} on {{timestamp}}.",
  "mockData": {
    "id":        "Sample id",
    "amount":    123.45,
    "region":    "Sample region",
    "timestamp": "2026-01-01"
  }
}

Field type → mock value mapping (deterministic fallback):

Field type Mock value
STRING "Sample <fieldName>"
DOUBLE / FLOAT 123.45
INTEGER / LONG 42
BOOLEAN true
DATE / TIMESTAMP "2026-01-01"
(any other) "<fieldName>-value"

Per-Job Configuration

Narrativization is configured on a per-job basis via the JobDTO model and Job Form UI:

{
  "id": "1",
  "name": "Default_Job",
  "repositoryConnector": "FileSystem_Local",
  "outputConnector": "PGVector_Output",
  "path": "/data",
  "narrativization": {
    "enabled": true,
    "template": "Document {{filename}} with size {{size}} bytes was scanned from {{path}}.",
    "connectorType": "filesystem"
  }
}

When a job starts, JobOrchestrator checks narrativization.enabled(). If true, it applies MustacheTransformationConnector(template) to each RepositoryDocument stream before publishing ingestion messages to Kafka.

AI Engine Configuration

The Copilot reads the active AI engine and Ollama chat model from properties:

spring:
  ai:
    copilot:
      engine: ollama   # default ('ollama' or 'openai')
    ollama:
      base-url: http://127.0.0.1:11434
      chat:
        options:
          model: ${SPRING_AI_OLLAMA_CHAT_MODEL:llama3.2}
          keep_alive: 1m

When engine: ollama is configured, TemplateGenerationCopilot looks for a bean named ollamaChatModel in the Spring context (registered automatically by spring-ai-ollama-spring-boot-starter).

When no matching ChatModel bean is available (e.g. Ollama is not running, no starter on classpath, or integration test environment), the service automatically falls back to deterministic template generation — guaranteeing a valid 200 OK response in all environments.

Schema SPI

Every RepositoryConnector now exposes an optional schema introspection method:

// in oc-core
public non-sealed interface RepositoryConnector extends Connector {
    Flux<RepositoryDocument> scan(String basePath);

    /** Returns the schema of documents produced by this connector. */
    default ConnectorSchema getSchema() {
        return ConnectorSchema.UNKNOWN;
    }
}

The ConnectorSchema record:

public record ConnectorSchema(
    String connectorType,
    List<SchemaField> fields
) {
    public static final ConnectorSchema UNKNOWN = new ConnectorSchema("unknown", List.of());

    public record SchemaField(String name, String type, String description) {}
}

Iceberg Schema Introspection

IcebergRepositoryConnector overrides getSchema() to return the live Iceberg table schema:

@Override
public ConnectorSchema getSchema() {
    // Reads the Iceberg table schema and maps each NestedField
    // to a SchemaField(name, typeString, docString)
    return new ConnectorSchema("iceberg", fields);
}

This allows the Copilot API to automatically populate field definitions when called in the context of an Iceberg-backed connector.

Integration Test

A dedicated test script exercises all 4 layers of the narrativization stack without requiring a full infrastructure setup:

# Run layers 1–3 (no Docker/database needed — CI-safe)
./scripts/test-narrativization.sh

# Run all layers including live HTTP test
./scripts/test-narrativization.sh --e2e

# Target a custom runtime host
./scripts/test-narrativization.sh --e2e --url http://staging.host:8080

Test Layers

Layer What is verified
Layer 1 — Mustache Engine MustacheTransformationConnectorTest runs the JUnit test; verifies "On 2026-07-23, region EU sold products: Laptop Monitor with amount $45000." output; checks JMustache 1.16 JAR in Maven cache
Layer 2 — Schema SPI ConnectorSchema + RepositoryConnector.getSchema() compile cleanly; IcebergRepositoryConnector.getSchema() override is present
Layer 3 — Copilot REST API NarrativizationCopilotIT (Mockito, no database): validates controller returns template + mockData, Ollama is the default engine, endpoint is registered at /api/transformation/copilot/generate, DTOs have the correct fields
Layer 4 — E2E curl Sends a live POST with 4 fields (id, amount, region, timestamp) and asserts the response JSON contains template referencing field names and correctly typed mockData values

Sample Data Printed During Tests

The script prints the exact input data and expected output before each test for full transparency:

▸ Mustache template
    On {{date}}, region {{region}} sold products: {{#products}}{{.}} {{/products}}with amount ${{amount}}.
▸ Document input data
    metadata = { date: ["2026-07-23"], region: ["EU"], amount: ["45000"], products: ["Laptop", "Monitor"] }
▸ Expected output
    On 2026-07-23, region EU sold products: Laptop Monitor with amount $45000.

See Also

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally