Skip to content

Home

Jump to bottom
Rajat Goel edited this page Aug 9, 2026 · 3 revisions

Why odcs-java-sdk?

The Open Data Contract Standard (ODCS) has gained significant momentum for defining schemas, data quality assertions, and service-level agreements. However, the tooling landscape surrounding ODCS is heavily biased toward the Python ecosystem.

odcs-java-sdk is built to solve this JVM gap.

It provides a native, high-performance Java library that handles ODCS contract parsing, spec validation, version migration, and active runtime enforcement. Instead of relying on external process wrappers, data engineers can use a lightweight, zero-overhead Java library to enforce data contracts directly inside Spark jobs, Kafka event streams, and JVM services.

Core Capabilities by Module

1. odcs-core

The foundational engine with zero heavy runtime dependencies.

  • Parsers and Writers: Reads and outputs ODCS v3 specs in both YAML and JSON.
  • Spec Validator: Checks contracts against official JSON schemas and structural invariants.
  • Fluent Builder API: Provides a clean, type-safe Java builder to construct contracts programmatically.
  • Version Migration Engine: Uses a strategy pattern to upgrade older contracts (e.g., v2.x to v3.x) while generating audit logs for applied defaults and deprecated fields.

2. odcs-spark

Plugs directly into Apache Spark pipelines across Java, Scala, and PySpark environments.

  • Distributed Rule Execution: Compiles ODCS quality rules directly into native Spark SQL expressions (Dataset) that run in parallel across worker nodes.
  • Row-Level Quarantine (DLQ): Evaluates records row-by-row, splitting DataFrames into valid records and quarantined records (routed to a DLQ storage location with violation metadata).
  • Write Guardrails: Blocks bad writes before they hit Delta Lake, Apache Iceberg, or Apache Hudi production tables.

3. odcs-converters

Removes the friction of manual contract creation by translating existing metadata formats into valid ODCS v3 definitions.

  • Supported Formats: Apache Avro (.avsc), Protocol Buffers (.proto), JSON Schema, dbt schema.yml, and ANSI/PostgreSQL DDL.

4. odcs-kafka

A streaming enforcement module for event-driven architectures.

  • Interceptors and SerDe: Validates record payloads against ODCS specs before serialization on Kafka brokers.
  • Dead Letter Topics (DLT): Automatically routes non-compliant messages to a DLT with attached violation reasons.

5. odcs-catalog-connectors

Bridges contract runtime state back into governance and observability platforms.

  • Catalog Sync: Pushes ODCS contracts and execution status into OpenMetadata, DataHub, and Atlan.
  • OpenTelemetry Exporter: Emits validation metrics (odcs.validation.passed_rows, odcs.validation.failed_rows) to Prometheus, Grafana, or Datadog.

Current Status and Roadmap

Module Status Primary Scope / Capabilities
odcs-core 🟢 Released (0.1.x) Core POJOs, YAML/JSON parser and writer, spec validator, fluent builder API, and version migration engine.
odcs-spark In Active Development Distributed Spark SQL rule compiler, DataFrame validation, write guardrails, and row-level DLQ quarantine routing.
odcs-converters Planned (Phase 2) Automated schema conversion from Avro, Protobuf, JSON Schema, dbt models, and SQL DDL into ODCS specs.
odcs-kafka Planned (Phase 3) Streaming SerDe validation, record interceptors, and automated Dead-Letter Topic (DLT) routing.
odcs-catalog-connectors Planned (Phase 3) Catalog sync with OpenMetadata, DataHub, Atlan, and OpenTelemetry (OTel) metric reporting.
Scarf Pixel

Clone this wiki locally