Migration Guide

ManifoldCF Migration Guide (oc-mcf-migrator) 🔄

OpenCrawling provides an enterprise migration tool, oc-mcf-migrator, designed to translate an existing Apache ManifoldCF crawler configuration — repository, output, and transformation connections, as well as scheduled jobs — directly into OpenCrawling.

It lives as a first-class module in the OpenCrawling reactor (oc-mcf-migrator/) and is exposed through three interfaces:

1. Command-Line Interface: Available standalone (java -jar oc-mcf-migrator-*-cli.jar) or as part of the unified CLI (oc mcf via oc-cli).
2. Web Migration Wizard: Visual step-by-step wizard in oc-admin-ui (Sidebar -> ManifoldCF Migration).
3. REST API: Programmatic endpoints on oc-runtime (POST /api/mcf-migration/plan and POST /api/mcf-migration/apply).

Important

Strict Mapping Principle: Anything with no direct connector-level mapping is skipped and clearly reported with actionable next steps (recommendedAction), never guessed or silently dropped.

---

🏗️ Architecture & Component Integration

flowchart TD
    subgraph Sources["ManifoldCF Sources"]
        LiveAPI["Live REST API\n(--mcf-url)"]
        JsonFiles["JSON Snapshots\n(--mcf-input-dir)"]
        XmlExport["Native XML Export\n(--mcf-export-file)"]
    end

    subgraph CoreEngine["oc-mcf-migrator (Core Engine)"]
        Parser["Entity Parsers & Adapters\n(DocumentSpecificationParser, McfXmlToJsonAdapter)"]
        Registry["ConnectorMapperRegistry\n(File, Vespa, OpenSearch Mappers)"]
        Engine["MigrationEngine\n(Plan, Fidelity Notes, Compatibility Score)"]
        ReportGen["Report Renderers\n(Markdown, JSON, HTML)"]
    end

    subgraph ConsumerInterfaces["Consumer Interfaces"]
        CLI["OpenCrawling CLI\n(oc mcf audit | import | convert)"]
        REST["oc-runtime REST API\n(/api/mcf-migration/*)"]
        UI["oc-admin-ui\n(Migration Wizard)"]
    end

    subgraph Targets["OpenCrawling Targets"]
        LiveRuntime["Live OpenCrawling Instance\n(POST /api/connectors, POST /api/jobs)"]
        OISFiles["OIS Job Files\n(ois/v1alpha1 YAML/JSON)"]
    end

    LiveAPI --> Parser
    JsonFiles --> Parser
    XmlExport --> Parser
    Parser --> Engine
    Registry --> Engine
    Engine --> ReportGen
    Engine --> CLI
    Engine --> REST
    REST --> UI

    CLI -->|import| LiveRuntime
    CLI -->|convert| OISFiles
    UI -->|Apply| LiveRuntime

Loading

---

🧭 Subcommands & Execution Modes

Command	Action	Writes to OpenCrawling?	Output
list-mappers	Discovers and prints registered connector mappers	Never	Console table of supported ManifoldCF classes and target types
audit	Extracts, maps, assesses fidelity, and reports	Never (dry-run)	Detailed audit report (markdown, json, or html)
import	Extracts, maps, and writes supported connections and jobs	Always (live write)	Creates/updates connectors and jobs via OpenCrawling REST API
convert	Extracts, maps, and emits job specifications	Never (pure file-to-file)	Standalone Open Ingestion Standard (OIS) (ois/v1alpha1) YAML/JSON files

Example CLI Invocations

# 1. Discover registered connector mappers
oc mcf list-mappers

# 2. Audit an existing ManifoldCF instance (dry-run report)
oc mcf audit \
  --mcf-url http://localhost:8345/mcf-api-service/json \
  --oc-url http://localhost:8080 \
  --report-format html \
  --report-file ./migration-report.html

# 3. Import supported connections and jobs directly into OpenCrawling
oc mcf import \
  --mcf-url http://localhost:8345/mcf-api-service/json \
  --oc-url http://localhost:8080

# 4. Convert ManifoldCF jobs to Open Ingestion Standard (OIS) YAML files
oc mcf convert \
  --mcf-input-dir ./mcf-snapshots \
  --output-dir ./ois-jobs \
  --output-format yaml

---

📥 Input Source Options

oc-mcf-migrator can extract ManifoldCF configurations from three distinct sources:

1. Live REST API (Default):
   • Flags: --mcf-url, --mcf-user, --mcf-password (or env vars MANIFOLDCF_URL, MANIFOLDCF_PASSWORD).
   • Reads /json/repositoryconnections, /json/outputconnections, /json/transformationconnections, /json/authorityconnections, and /json/jobs.
2. Saved JSON Snapshots (--mcf-input-dir):
   • Reads from a directory containing the raw JSON payloads saved from the 5 endpoints above (repositoryconnections[.json], outputconnections[.json], transformationconnections[.json], authorityconnections[.json], jobs[.json]).
   • Ideal for air-gapped environments or offline testing.
3. Native XML Export (--mcf-export-file):
   • Reads ManifoldCF's combined configuration XML export created by its native ExportConfiguration utility.
   • Enables zero-network, direct file-based migration without requiring a running ManifoldCF service.

---

🔧 Connector Mapping & Redirection

Automatic Connector Mapping

The engine discovers registered ConnectorMapper implementations using Java SPI (ServiceLoader). Out of the box, the following mappers are included:

• Filesystem Connector: Maps org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector to OpenCrawling filesystem-source.
• Vespa Output Connector: Maps org.apache.manifoldcf.agents.output.vespa.VespaOutputConnector to OpenCrawling vespa.
• ElasticSearch / OpenSearch Connector: Maps org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchConnector to OpenCrawling opensearch2.

Manual Redirection (--map-connector)

If a ManifoldCF job references a connection without a direct automatic mapper (e.g. an existing Solr connection being directed to a new Qdrant vector store in OpenCrawling):

oc mcf audit \
  --mcf-input-dir ./mcf-snapshots \
  --map-connector "Solr_Output=Qdrant_Vector_Store"

The source connection is not created, but any job referencing Solr_Output has its target replaced with Qdrant_Vector_Store and is treated as supported.

Scoping Migrations

To migrate only specific connections or jobs, use:

• --only-connections "ConnA,ConnB"
• --only-jobs "Job1,Job2"

---

🌐 Web Migration Wizard (oc-admin-ui)

In addition to the CLI, OpenCrawling provides a graphical wizard within the web administration console:

1. Configure: Select the ManifoldCF source (Live API or JSON snapshot directory) and specify credentials.
2. Extract & Plan: The backend invokes POST /api/mcf-migration/plan to analyze all connections and jobs, computing a job-based compatibility score.
3. Review & Scope: Review supported vs. unsupported components, inspect fidelity notes, and deselect specific items if desired.
4. Apply: Invokes POST /api/mcf-migration/apply to register connectors and jobs in the OpenCrawling runtime.
5. Results: Review creation status and trigger immediate job execution via the OpenCrawling scheduler.

---

📄 Reports & Fidelity Tracking

All subcommands produce comprehensive reports in your choice of format (--report-format markdown|json|html):

• Compatibility Score: Job-based percentage indicating the proportion of jobs that can successfully execute in OpenCrawling.
• Field Fidelity Notes: Detailed notes indicating fields that were defaulted, transformed, or dropped due to architectural differences (e.g., ManifoldCF schedule patterns vs. OIS cron strings).
• Recommended Actions: Clear guidance on manual steps needed for dropped or high-risk configuration fields.
• Secret Redaction: Passwords, tokens, and sensitive credential fields are automatically masked in all generated reports.