Maven Connector Archetypes

Maven Connector Archetypes Suite 📦

OpenCrawling provides an official suite of Maven Archetypes under org.opencrawling.archetypes allowing developers and ecosystem partners to instantly scaffold ready-to-build, standardized custom connectors with a single mvn archetype:generate command.

---

🛠️ Available Archetypes

Archetype Artifact ID	Connector Type	Description & SPI Interface
opencrawling-archetype-repository-connector	Repository Connector	Ingestion sources (CMS, REST APIs, Databases, File Systems). Implements RepositoryConnector (Flux<RepositoryDocument>).
opencrawling-archetype-output-connector	Output Connector	Ingestion destinations (Vector DBs, OpenSearch, Elasticsearch, S3). Implements OutputConnector (Mono<Void>).
opencrawling-archetype-transformation-connector	Transformation Connector	Data processing, chunking, and AI enrichment. Implements TransformationConnector (Flux<RepositoryDocument>).

---

🚀 Quickstart CLI Commands

1. Generate a Repository Connector (Ingestion Source)

mvn archetype:generate \
  -DarchetypeGroupId=org.opencrawling.archetypes \
  -DarchetypeArtifactId=opencrawling-archetype-repository-connector \
  -DarchetypeVersion=1.0.0-SNAPSHOT \
  -DgroupId=com.mycompany.connectors \
  -DartifactId=my-repository-connector \
  -Dversion=1.0.0-SNAPSHOT \
  -DconnectorName=SampleRepositoryConnector

2. Generate an Output Connector (Vector Store / Destination)

mvn archetype:generate \
  -DarchetypeGroupId=org.opencrawling.archetypes \
  -DarchetypeArtifactId=opencrawling-archetype-output-connector \
  -DarchetypeVersion=1.0.0-SNAPSHOT \
  -DgroupId=com.mycompany.connectors \
  -DartifactId=my-output-connector \
  -Dversion=1.0.0-SNAPSHOT \
  -DconnectorName=SampleOutputConnector

3. Generate a Transformation Connector (Data Processing / Enrichment)

mvn archetype:generate \
  -DarchetypeGroupId=org.opencrawling.archetypes \
  -DarchetypeArtifactId=opencrawling-archetype-transformation-connector \
  -DarchetypeVersion=1.0.0-SNAPSHOT \
  -DgroupId=com.mycompany.connectors \
  -DartifactId=my-transformation-connector \
  -Dversion=1.0.0-SNAPSHOT \
  -DconnectorName=SampleTransformationConnector

---

🐳 Docker Compose Overlay & Admin UI Testing

Every generated archetype comes pre-configured with:

• Unit & Integration Test Suite: Uses JUnit 5, Reactor StepVerifier, and maven-failsafe-plugin (*IT.java execution via mvn verify).
• Official OpenCrawling Docker Composition (docker/docker-compose.dist.yml): Launches OpenCrawling backend, Admin UI (http://localhost:3000), Postgres + pgvector, Redis Stack, and Kafka.
• Docker Compose Overlay (docker/docker-compose.override.yml): Mounts the custom compiled connector JAR into the running OpenCrawling container volume (/app/plugins/) and pre-seeds the Admin UI (/data/connectors.json).

Testing & Verification Workflow

# 1. Package custom connector JAR
mvn clean package

# 2. Start OpenCrawling Docker environment with custom connector and Admin UI overlay
docker compose -f docker/docker-compose.dist.yml -f docker/docker-compose.override.yml up -d

# 3. Interactively configure, run, and monitor jobs in Admin UI:
# Open http://localhost:3000 in your browser

# 4. Execute unit and integration tests
mvn verify

# 5. Tear down Docker environment
docker compose -f docker/docker-compose.dist.yml -f docker/docker-compose.override.yml down